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ABSTRACT 


The relationship between page size, program behavior, 
and page fetch frequ2ancy in storage hierarchy systems is 
formalized and analyzed. It is proven that there exist 
cyclic program reference patterns that can cause page fetch 
frequency to increase significantly if the page size used is 
iacceased (e.g., reduced by half). Furthermore, it is 
proven in Theorem 3 that the limit to this increase is a 
linegac function of primary store size. Thus, for example, 
On a typical current-day paging system with a large primary 
store, the number of page fetches encountered during the 
@axecution of a program could increase 200-fold if the page 
Slze were reduced by naalf. 


The concept of temporal locality versus spatial 
locality is postuiated to explain the relationship between 
page size and program behavior in actual systeaus. This 
concept is used to develop a technique calied the 
“tupl2-coupling" approach. It is proven in Theorem 5 that 
when used in conjunstion with conventional hierarchical 
storage system replacement algorithas, tuple-coupling yields 
the benefits of smaller page sizes without the dangers of 
2xplosive page fetch activity. 


consistent with the results above and by generalizing 
conventional two-leval storage systems, a design for a 
general multiple lavel storage hierarchy systen is 
presented, Particular algorithms and implementation 
techniques to be used are discussed. 
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CHAPTER 1, 


INTRODUCTION AND PLAN OF THESIS 


1.0 Introduction 

The primary goal of this thesis is to provide insight 
into and shed additional light on several key problegs in 
the design and analysis of general storage hierarchy 


systeas, 


The importance of research in stcrage iierarchy systems 
has b2en pointed out by Prof. F. J. Corbaté recently in the 


MIT Project MAC Progr2ss Report VIII (July 1971): 


"By now, it has become accepted lcre in the computer 
System field that use of automatic management 
algorithas for memory systens, constructed of 
s2veral levels with different access times, can 
provide a significant simplification ot programming 
effort. ..+. Unfortunately, behind the mask ot 
asceptance hides a worrisome lack of knowledge 
b2hind how to engineer a multilevel memory systen 
with appropriate alyorithms which are matched to the 
load and hardware characteristics." 


Qn multiple level storage hierarchies, Prot. Jd. H. 
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Saltzer was even nore explicit {subject notes on 


"Information Systems", MIT, 1972, pe. 4-58): 


"An interestiny problem arises if one has three or 
more technologies to deal with. ... The problem of 


predicting the performance of a three level, 
automatically managed system is not at all well 
understood. ... Although the need for more than one 


level has already been argued, there is currently no 
kaown criterion for introducing three, four, or N 
levels for a given system. ... Although there are by 
now many implemantations of two level memory 
systems, the dynaaic management of a three or more 
level memory syst2ma is such an uncharted area that 
there do not yet exist examples of practical 
algorithmws which one can examine." 


[The specific goals and acccmplishments of this thesis, 
waich are further elaporated later, are: 

® Analyze the affect of certain parameters, such as 
page size, upon the performance of a storage 
systen,. 

°® Develop a concept of locality based upon oboth 
spatial and temporal adjacency in address 
reference patterns that explains certain anomalies 
discovered in actual paging systems. 

* Propose, formalize, and measure the performance of 
new "spatial-removal" storage Hanagement 
algorithms, iu particular "tuple-coupiing". 


* Design a practical algorithna for eftective 
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management of multiple level storage hierarchy 
systems ani demonstrate its effectiveness under 


some Simulated system loads, 


1,3 G2neral Structure of Thesis 


The key plan of this thesis is to investigate several 
ccucial problems and requirements of multiple level storage 
hierarchy systems. Particular areas are identified and 
corresponding theori2s developed and proven. A new and 
jg2neral design for storage hierarchy systems is aiso 
presented and evaluated. Finally, empirical measurements are 
presented to validate and calibrate the overall design and 


specific theoretical conjectures, 


This thesis is organizationally divided into 8 
chapters. The structure can be best introduced by outlining 


the content of the following chapters in the sections below. 


123.1 Chapter 2: Motivation for Storage Hierarchy Systems 


[This chapter presents a perspective on the storage 
hierarchy problem and the motivation for Such systems. It 
is primarily written for the benefit of people knowledgeable 
in th2 general computer field but who are not especially 


experienced in storage hierarchy systems. For the expert 
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cealer, this chapter exposes the biases and orientations of 
the author and thus sets the tone for the remainder of the 
thesis. This chapter also briefly reviews the history of 


cC2searcch in storage systems and cites numerous references, 


1.3,2 Chapter 3: Formalization of Storage Hierarchy Systems 


A description and formalization of the basic 
Sharacteristics of storaye hierarchy systems is presented in 
this chapter, This is followed by a summary and critical 
amalysis of research that directly relates to the specific 


joals of this thesis, 


1.3.3 Chapter 4: A Storage Hierarchy System 


In this chapter the key concepts of the proposed 
stocage hierarchy system are presented and discussed, The 


principle and novel techniques are briefly described below: 


1.3.3.1 Continuous Hierarchy 


[he ratio otf pearformance between adjacent levels is 
kept moderate (e.g., a factor of 100 or less) to minimize 
discontinuities or awkward special-case algorithms. This is 
ia contrast to many current systems with inter-level ratios 


of 1090 or more, 
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1.3.3.2 Shadow Storage and Page Splitting 


Information is transferred in decreasing smaller size 
blocks as it is passei up from low performance levels of the 
hierarchy toward the “request generator" at the uppermost 
level. Thus, the information that is finally received by the 
raquest generator has left a "shadow" behind in the lower 
levels. The significaase and rationale for this technique is 


further elaborated in Chapter 6, 


1.3.3.3 Automatic Management 


In order to reduce the load on the central processor 
and provide for more 2afficient and parallel ocperations, the 
storaje management function will be distributed and 
Lacorporated into tha storage levels (e.g., “intelligent” 
d2vic2 controllers [1], etc.). This technique also reduces 


the Complexity of the operating system software. 


1.3.3.4 Direct Transfer 


Storage transfers between two adjacent levels need not 


have any effect upon aor require the assistance ot any cther 


lavels (e.g, there is no need to move information from 
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level n to level 1 and then frem level 1 to level n-1 if 
only level n to level n-1 was needed; this two step process 
is often ceguired on contemporary systems). Direct transfer 
is accomplishei by synchronizing non-mechanical storage 
devices or by using “"rubber-band" buffers [33] between 


alectco-mechanical storage devices, 
1.3.3.5 Read Through 


Storage transfers, as noted above, are only nade 
batwean adjacent levels of the hierarchy, such as from level 
no to level n-1. But, each level, such as Level n-1, can 
connect its input bus (from lower level n) to its output bus 
(to bigher level n-2) so that the data can be read through 
(i.2., transterred to level a-2 while being stored in ievel 
n-1). A Similar, though specialized, techniygue is already 
usei in certain systems, such as the IBM Systea/370 Models 


135 and 165 cache systems [52]. 


This results in performance Simiiar to a direct 
connection from each level to the request generator but it 
proviies much more control in the storage levels and a much 


Simpler structure, 
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1.3.3.6 Store Behind 


By uSing the excess capacity of the inter-ievel 
Caannels, there is a continual tlow of altered data from tne 
higher levels to the Lowest level permanent storage. Thus, 
tae actual updated information is stored behind (after) the 
store initiation fron the request generator. The updated 
1aforaation is propagated down, level by level. Whenever 


1laforaation is alterei at a particular level, it is tayged 


as altered and is schaduled for a "store behind" operation. 
1.3.4 Chapter 5: Analysis of Page Size Considerations 


Jne of the most important parameters of a storage 
hierarchy system is the page size chosen as the unit of 
transfer between two levels of the hierarchy. In this 
caaptear, the factors influencing page size are examined from 
the device charactaristics viewpoint and the progran 


op2havior viewpoint, 


Jf particular concern, it has been noticed by Hattield 


{47] and Seligman { 78] and formalized in Chapter 5 that: 


"There exists a page trace, P, and demand-tetch 
FIFO-removal or L&U-removal inter-level storage 
systems, S and S*, with page sizes N and N‘=N/2, 
Caspectively, such that the ratio, r, of fetch 
frequency f' to f exceeds 2," 
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This result runs counter te the hoped for behavior of 
decreased page sizes as noted by Denning [25]}: 

" ye. Stall pages permit a great deal of compression 

without loss of 2fficiency. Small page sizes will 


yield Significant improvements in storage 
utilization ... " 


In this chapter the significance of this problem is 
iamonstrated by proving that even "well-behaved" removal 
alyorithms, such as stack algorithms [63], are not immune to 
this adverse performance behavior. Furthermore, the nature 
ot this phenomenon is analyzed and bounds on its behavior 


ace developed. 


1.3.5 Chapter 6: Spatial vs. Temporal Locality Modei of 


Program Behavior 


A primary rationale for hierarchical storage systems is 
based upon the "Principle of Locality". Unfortunately, this 
pcoinciple is stili a poorly understood, or at least 
controversial, phenomenon. It is difficult to determine the 
ocvigiaal "discoverec" of this principle but it is 
iataresting to note that its definition has changed in time. 
For 2xample, Denning (29, p.3], in 1968 loosely described 


locality as: 
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"the idea that a computation will, during an” 
interval of time, favor a subset of the information 
available to it," 
Later, in 1970, Denning [26, p.180] defined it more 
precisely based upon the concepts of "working set" and 
"reference density", which for a page i at time k: 
a(i,k) = Pr[reference r(k)=i], 
Sica that &(k) is tae ranking of all n pages based upon 
a(i,k); thus: 
SPRINCIPLE OF LOCALITY: The rankings R(k) are 
strict and the expected ranking lifetimes long." 
This is a much more rcestrictive definition of locality than 


ais earlier general concept. 


In fact, many current storage management systems were 
devised first, a general model was then constructed to 
dascribe the system, and finally a "formal" definiticn of 
locality was developed to be consistent with the storage 
management model. This is a reasonable strategy as long as 
tae underlying concepts of "the principle of locality" are 
not lost ain the procass. Unfortunately, this appears to 
nav2 happened on several occasions, In particular, most 
popalac definitions of locality tend to be useless tor 
analyzing or explaininj either the relationship ot page size 


upon program behavior or the impact of generalizing from 
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two-level storage systems to sultiple ievel hierarchical 


storage systems. 


In this chapter a new view cf locality is presented (or 
an Old-view resurrect2i since it most closely vesembies some 
9€ th2 very early descriptions of locality). In particular, 
it 1S shown that th2 general concept of locality can be 
Subdivided into two separate factors, temporal locality and 


spatial locality. Th2se concepts are defined and justified 


and then used to explain some peculiar phenomena 


(Sanomalies") observei in actual two-level storage systems. 


By means of address traces and storage systen 
SLmplifications, the t2mporal and spatial jiocality behavior 
of actual programs is emperically measured. These results 
are used to reinforc2 anid calibrate the storage hierarchy 


systea design presented in Chapter 4, 


1.3.0 Chapter 7: Spatial Removal Storage Management 


Algorithas 


Various hierarchy storage management algoritams, such 
as fetch (e.g., demand-fetch) and temporal removal (e.g., 
first-in first-out (FIFO), least recently used (LRU), etc.) 
have peen developed, primarily for two-level hierarchies. 


Therc@ appear to be no spatial removal algorithms described 
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ia th2 literature. Based upon Chapter 6, several spatial 


algorithms are proposed and analyzed. 


It is also shown taat some cf the problems aescribed in 
Chapter 5 can be solved by spatial removal algorithms. In 
particular, Hatfield °§48] noted that: 

"as yet we have been unable to prove that there is a 
ceplacement alyorithm usiny only the past history of 
page requests which cannot generate more than twice 
the exceptions with half size pages." 
In this chapter a new algorithm, named tuple-coupling, is 
presentei, It is formally proven that it satisfies 


Hatfield's reyuirements above, 


Furthermore, the operational behavior of tuple-cou;,ling 


1s analyzed by measuring the performance ot actual procgrams. 


1.3.7 Chapter 8: Discussion and Conclusions 


In addition to a yeneral summary of the significant 


aspects of the thesis, this charter also outlines iwpoctant 


aceas for future research. 
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CHAPTER 2. 


THE STORAGE HIERARCHY PROBLEM 


2.9 Introduction 

The evolution of computer systems has been marked by a 
continually increasiny demand for taster, larger, and more 
2conoaical storage facilities. In addition to the obvious 
concecn for better parformance, the organization of a 
computer system's storage plays a key role in = program 
development and programmer efficiency. It has often been 
claimed that "any software design blunder can be overcome by 


adding more memory", 


It has become generally recognized that the conriictingy 
requirements of hiyh-p2rftormance yet low-cost storaye may be 
b2st satisfied by 2 mixture of technologies combing 
axpeasive Aigh-pertormance devices with inexpensive 
lower-perforaance devices, This strategy has been given 
saveral names, Such as "hierarchical storage systea", 
"automatic multilevel storage management", "virtual nemory", 
and the inevitable "virtual memory system tor the automatic 
multilevel management of a hierarchy of storage devices", 


ln this thesis the somewhat shorter term storage hierarchy 
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systema will be used. 


Investigations into autcmated storage hierarchy 
techniques can be traced back more than a decade. It we 
w2r2 to include manual techniques, we would find storage 
hieracchies at the vary dawn ot the "computer age", 
Unfortunately, there are still many unsolved and poorly 
understood problems. [This situation can be partly explained 
py the fact that th2se systems tend to be (1) extremely 
compl2x, {2) ill-suited to most conventional analytical 
tecaigues, and (3) deeply influenced by the rapidly evolving 
computer technology which keeps “changing the ground ruies" 
at often frightening rates. in spite of these challenging 
stumbling blocks, a successful storaye hierarchy system is 
so important to the future usefulness of computer systems 


that we cannot afford to abandon the search. 


2.1 Storage Hierarchy sbjectives 


Before delving into details, it is worthwhile to 


briefly consider the oeeds and uses fcr an effective storaye 


hierarchy. 
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2.1.1 System Performance and Economics 


AS logic tecanology and computer architecture 
tacaniques have advaaced, we have found it possible to 
produce systems of inztredible speed. Such systems are often 
rated, rather crudely, in terms of MIPS (millions of 
instructions per second). Experimental system of over 100 
MIPS aave been developed (e.g., ILLIAC IV and CDC STAR). 
oven “conventional” large-scale systems have passed the 5 or 
1) MIPS mack (e@.g-, CDC 7600 and IBM 370/195). It has long 
been observed that the input/output (1/0) requirements, 
especially for “seconiary storage", of a conventional system 
tani to be strongly celated to the processor's speed. in 
fact, based upon several empirical measurements, it has been 
postulated that a computer system averages 1 bit of I/O for 
avery instruction executed (this is often referred to as 
Aadanl's Constant [ref j). As a result, many of these 
tiilgh-performance syst2as have been confronted with massive 
bottleneck problems in the I/O area, especially since these 
I/O demands tend to occur in bursts, An effective storage 
hierarchy system couli yo a long way toward reducing this 


problen. 


At the other end of the spectrum we tind that mediun- 
and low-cost processors, the latter are usually called 


Mini-computers, have nade substantial advances in recent 
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y2acs. The term "mini" can be quite misleading. Tnese 
processors are typically huudreds of times faster than the 
early commercial computers at a fraction of the cost (e.g., 
the UNIVAC I, circa 1951, could perform about 2000 12-digit 
additions per seconi whereas contemporary mini-computers 
Qperate at around 1,399,000 5-digit additions per second). 
Although these mini-processors may be midgets ccmpared to 
the computational problems attacked by their "big brcthers" 
dasctribed above, they are more than adequate for the vast 
majority of infomatioa processing problems which have modest 
computational requirements. Due to technological advances 
and economies ot scale resulting fron large-scale 
production, some minicomputers are available for less than 
32000 with slightly slower micro-computers being offered for 
as little as 366 [| 13]. In spite of these technoloyical 
alvances, these processors have not had much impact cn most 
laforaation system needs due to the continuing economic 
problam of producinjy large capacity inexpensive storaye 
aay ieee even at the modest pertormance reguired. A $66 
processor is largely irrelevant if the storage costs are in 
the $100,000 or mor2 range, By developing an effective 
storaye hierarchy system, we can go a long way toward 
beinging the storage costs down to the level ot these 
lnexpansive processors, AS a result, a tremendous number of 
currently known tachnical solutions to information 


processing problems will finally become economically 
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f2asible solutions, 


201.2 Simplity and Automate Programming 


As noted earlier, th2 organization of a computer's 
storage system has a considerable impact upon programa 
d2avelopment and programmer efficiency. To a large extent, 
this potential increase in preductivity is obtained by 
caducing or eliminating constraints normally imposed by the 
storage system, These constraints often distract the 
pcrojrammer to the extent that he devotes a substantial 
anount of his time to overcoming the system*s limitations 
cath2c than solving the intrinsic problems. Shooman [80] 
noted that: 

"The inherent error content of some programs is 
claimed to be related to the excess nemory capacity 
available, The theory here is that if the memory is 
very cramped, th2 software writers will have to 


rasort to overlays and other coding "tricks" toa 
syueeze the desiced functions into the allocated 


m2mOry space. It is assuged that these tricks 
latroduce great complexity and are the seat of many 
@rrors. This effect is cited by designers of 


alLrborne computers where the allocation of another 
block of 4k of memory is a major design decision." 


For example, the programmer often has to worry about: 
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2.1.2.1 Programming lanyuage code efficiency. 


Lf a higher-ieval language compiler tends to produce 
programs that are at all laryer than those produced by a 
low-level language translator, it may be necessary to use 
the low-level languayg2 to conserve storage. This constraint 
is contrary to the generally accepted fact that high-level 


languages enhance programminy prcductivity. 


2.1.2.2 Program size. 


For any specitic storaye size, there are programs that 
camnot be easily written to fit into that size constraint. 


Yat, proyrammers frequently try - with considerable effort. 


2e1.2.3 Data structures. 


The programmer is often faced with the need to choose 
batween a data structure representation that is convenient 
to use and another representation that "saves storage", 
Tais saving may xraquire the use of an awkward oc 


unnecessarily complex data structure representation. 
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21.2.4 Specitic equipment characteristics. 


If the programaer must get the "most" out of his 
storage system in teras of capacity and performance, he may 
r2sort to techniques that are peculiar to his’ specific 
storage system equipment. if the equipment is changed, 


there may be a considerable impact upon his software. 


We would like to. develop storage hierarchy technigues 
that eliminate, automate, or at least mininize the 


programming probleas iescribed above, 


2.1.3 Integrate New Technologies and Applications 


Although there has been continual evolution, the basic 
storage device techaslogies in commercial use have not 
changed dramatically in the past decade. As a result, tnere 
has b2en a tendency, motivated by actual need, to relate 
applications to the specific available technologies. This 
has caused certain application areas to be abandoned as 
"infeasible" and many storage management strategies to be 
discredited as "irrelavant" or "inefficient". In the passage 
of tine we remember the applications and tecaniques in use 
but frequently forget or ignore the alternatives possible 


and tne reasons for bypassing these alternatives. 
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After this rather long "rest", it appears that we are 
on the verge of some major “awakenings” in applications and 
tachnology. It is hard to quantify the new application 
n2zeds other than reguiring more and faster storaye for less 
money. Section 1.1.2 presents some of these motivations, the 
cevitalized interests in time-sharing, artificial 
ilnatelligence and automatic programming are also "fanning the 


tire", 


Due to the uncertainty of advanced research in storaye 
davica technologies, it is difficult to torsee accurately 
which of the many active efforts will succeed (see for 
axample, Ayling [7], Best [15], Bobeck [16], Camras [17], 
D21l °24j, Fields [35], Gardner [39], Howard [50], Matick 
[&Matick.], Myers (59]j, Rector [74], Shahbender {79], 
Thoapson [85]). Considering the technical advances clearly 
demoustrated in the laboratory and the driving "protit" 
motivation, it is reasonable to expect some dramatic changes 
ia th2 next few years. Even if we don't know what or when, 


w2 would be foolhearty to totally ignore this situaticn,. 


Table 1 below andicates the performance and _ price 
characteristics of typical current-day storage technologies. 
[The t#o entries marked by questicn marks (?), Bulk Store and 


Giant Store, indicat2 new technologies that have already 


Davic2 


1. 


2e 


De 


Oe 


Cache Store 
(1BM 3165) 


Store 
3369) 


Main 
{IgM 


Bulk 
(AMS 


Store? 


Lacye Store 


(IBM 2305-2) 


Mass Store 
(IBM 3339) 


giant Store? 


(Gcumman 
MASSTAPE) 


SSUL 35 }) 


Storaye 


Randoa Maximum 

Acc2ss Transfer 

Time Rate 

dseconds) (byte/sec) 

1.6x107-7 1x 104 

(16) as) (100M b/s) 

1.49x10-6 1,6x107 

(1.44 us) (16M b/s) 

1, 3x10-* 8x 106 

(130 us) (8M b/s) 

5x10-3 1.5x106 

(5 ms) (1.5M b/s) 

3.8x10-2 8x 105 

(38 ms) (800K b/s) 

6x1) 6x 105 

(6 sec) (600K b/s) 
Table 1. 
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Price 


{$/byte) 


8. 8x100 
($8.80) 


5x10-2 
(562) 


8.8x10-2 
(8.82) 


2.2x 10-2 
(2. 2¢) 


4.5x10-* 
(.045¢) 


222x10-5 
(.0022¢) 
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(£t1) 


Capacity 
(bytes) 


1,.6x10% 
(16K) 


5212x105 
(512K) 


2x 106 
(2M) 


1.1x107 
(114) 


2x 108 
(200M) 


1.6x1019 
(168) 
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p2en placed in limited use. Since these two 
cost/performance positions were net part of our 
"traditional" technologies, we are faced with the prcblem of 
possible modifying ouc applications and developing new 
strategies to efficiently, effectively, and, hopefully 
Optimally, integrate them into our overall hierarchical 


storage systear, 


As the entire spectrum of computer architectures, as 
w2ll as storage device technologies, undergoes reshuttlings, 
both 2volution as well as revolutions, it 1s worthwhile to 
caview and reconsider our current concepts on storaye system 
ji2sign. fanle 1, although a Simplified summary of current 
storage technologies, illustrates the fact that there exists 
a Spectrum of devices that span about 6 orders of magnitude 
of price/performance (100,000,000%) . This is quite 
Sigaificant in the light of the excitement that normally 
accogapanies an improvement of 10-20% in performance or a 
decrease of 10-204 in price in current-day systems. The 
participants in this "storage sweepstakes" may Change in 
time, but with such large price/performance stakes, there 


will be continuing benefits to "playing the game" better. 
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2.1.4 Understanding of Prograua and System Behavior 


As noted earlier, the detailed operational behavior of 
computer systems is often extremely complex, Thus, 
decisions on hardwar2, sottware, and system design must 
often be made in spit2 of insufficient knowledge. A better 
undarstandiny of program and system behavior is essential to 


the intelligent and efficient development of future systems. 


{It is hoped that the research tc ke conducted as part 
of this thesis will shed considerable light cn these 


matters. 


ee Se 


"Storage hierarchy System" and similar terms have been 
used in many contexts. Consistent with the objectives 
Sutlined in the pr2vious section, certain particular 


contexts are assumed in this research, 


2.2.1 Spectrum of Approaches 


The problems of storaye hierarchy management have been 


attacked by a host ot approaches, We can loosely 


characterize these efforts into three categories: 
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2.2.1.1 Manual Hierarchy Management 


3iven a specific enseable of storage device 
t2chnologies, after considerable thought the programmer can 
@xplicsitly or implicitly specify how his intormaticn (1.e., 
poojgrams and data) shouid be organized and distributed 
Within the hierarchy and how and when his information should 
be re-arranged. Having determined the distribution, he must 


also specify his access to specify information accordingly. 


When a programmer is directly operating upon his 
information at tne lowest level (e.g., using machine 
language, direct I/)9 requests, etc.), he is explicitly 
controlling the storage hierarchy, this is explicit manual 


a oe vn 


uLecacchy management. In most conventional systems, the 
programmer communicates with the system via proyramM@inyg 
languages and control cards, Although this can relieve much 
of th2 tedious or intricate details of storage managewent, 
the overall control of the storage hierarchy is still 
prizacily the responsibility of the fproyrammer. This is 
laplisit manual hierarchy management. | 

Yanual storage management can be very economical since 


it usually requires nd special hardware features nor special 


system software. Furthermore, it places the control of the 
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storage hierarchy in the hands of the programmer who is 
presunably tne one most familiar with his needs. Manual 
storajye manayement, ia its many manifestations, is the most 


common storaye hierarchy approach in use today. 


Manual storaye management has many disadvantages, 
though. The amount ot detail that the programmer must 
understand and use can add significant complexity to this 
task. This then intecoduces additional areas of error and 
decreased productivity. Furthermore, the assumption that 
the programmer is the best judge of optimal storage 
otganizatioa is often wrong. The complexities and dynamics 
common to modern systems are often beyond the understanding 


of most application programmers, 


Multiprogramming, an almost universal technigue in 
current systens, necessitates strategies for giobal 
Qptimization which usually differ substantially from the 
iadividual Local optimizations of each progran. For these 


reasons there has been continual search for "a better way". 


2.2.1.2 Semi-Automatic Hierarchy Management 


Many techniyues nave been developed to minimgize the 


amount of effort reyguired of the proyrammer and to provide 


teedback to hin, Ta2 programmer still has the ultimate 
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control in such a seni 


syster, 


certain of these techniques are based upon tne concept 
of th2 programmer providing "hints" to the system. these 
hints form the basis tor a partially automated, partially 
manual storaye management system. Although not especially 
widespread, this approach has been used in several systems 


(2.g-, Jensen et al [53], O'Neill et al [70], etc.). 


If there is a single application that is quite large 
and compiex, techniyues have been developed to analyze the 
actual performance and provide feedback to the programmer, 
This approach is primarily used in specialized, dedicated, 
predictable, high-performance systems, such as an airline 
reservations system, Numerous attempts have been reported, 


such as Arora et al {3]J, Kamamoorthy et al [72], etc. 


The various semi-automatic hierarchy Managenent 
approaches help to raduce the programmer's effort and to 
attain a better local optimization. Although useful tor 
c2artain applications, these strategies do not remove’ the 
disadvantayes already aoted with manual hierarchy management 


sy steas, 
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2e2e1.3 Automatics Hierarchy Management 


certain aspects of logical information organization are 
inherent in a programmer's basic alyorithm. In an automatic 


hLegacchy management system, all aspects of the physical 
information organization and distribution that are 
icrelavant to the unierlying logical structure Should be 
ceamoved from the programmer's responsibility. The 
programmer May wish to, maybe even be encouraged to, use 
algocithms tnat are known to perform well in conjunction 
with the automated hierarchy management. But, the central 
caspoasibility of the storage hierarchy management is 


ramoved from the programmer, 


Since this approach darectly tocuses on the storage 
hierarchy objectives presented earlier, it will be the 


primacy approach to be pursued in this thesis, 


2.2.2 Spectrum of Analysis Efforts 


Each of the storage hierarchy approaches mentioned 
above, primarily semi-automatic and automatic, have been 
subjected to various torms of analysis. In this section we 


briefly outline the principal aeficiencies of these efforts. 
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2.2.2.1 Generalized Models 


Ine popular form of analysis is to assume a generalized 
aod2l for hardware, sottware, and system behavior. If one is 
careful in caoosing tne Characteristics of the model (e.g., 
Poisson arrival and sarvice times, etc.), it is possible to 
davelop precise analytical solutions. Unfortunately, it is 
usually difficult to validate these models except for rather 
simple solutions. Furthermore, since there are tew truly 
aatomatic storage hierarchy systems in general use, it is 
extremely difficult to even determine realistic parameters 


for these generalized models even'if the models were valid. 


Generalizea mod2lis have keen reported in several 
papers, such as Aho et al [2] and Denning [25] in the 


Bibliodgraphy, 


2626262 Constrained Models 


Another variation on the generalized model scheme is to 
analyze a particular program and then model its relationship 
to the rest of the system. There are at least two 
Shortcomings in this approach, First, as in the yeneralized 
model case, it is iifficult to realistically model the 
celationshnip between 4a program and the rest of the systen. 


Second, the analysis and measurement of the particular 
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peogram is normally converted into scme form of probability 
watcix Or probabalistic reference pattern. In either case, 
Significant effort is reguired to accurately measure the 
program's behavior. Furthermore, the probabalistic 
Snharacteristics are usually aggregated to reflect the 
overall behavior of the program and, asa result, the 
dynamic nature of the program and its impact on the storage 


hierarchy are often lost. 


Example analyses of constrained models can be found in 
ceferences: arora and Gallo {5]jJ, Hatfield and Gerald [47], 


L2wis and Yue [60], and Ramamoorthy and Chandy {72]. 


2.2.2.3 Limited Environment 


A common deficiency ot most previcus research is that 
only a limited enviconment was considered, in particular 
automatic hierarchy management over cnly two levels uSing a 
single page size. Of course, most current-day computers have 
only amployed automatic hierarchy management in either Cache 
Systems (cache store - main Store) or Paging Systems (main 
Stor2 - Large store), Unfortunately, there is definite 
c2asons to believe that many of the conclusions and 
tecanigues demonstrated for a two-level hierarchy do not 
necessarily generalize to handle the spectrum ot program 


datail and device characteristics encountered ina truly 
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multiple level storage hierarchy. Furthermore, many or the 
papers that attempted to investigate general storage 
nieracchies assumed techniques and approaches that are 


prigacily based upon two-level hierarchy assumptions. 


This limited environment has been studied by numerous 
people, such as Aho et al [2], Helady etal [{10,11,12], 
Coffman and Varian {19,86], Conti et al {21,22}, Denning 
_25j, Fotheringham {; 33], Guertin [45], Kilburn et al 157], 


Mattson et al [63], Seligman [78], Smith [81], and Wiikes 


[88]. 


2.2.2.4 General Hierarchy Environment 


The studies of limited two-level storage hierarchies 
have been quite successful in many actual systems. A 
reasonable strategy would be to extend these technigues to a 
more general storaye hierarchy environment. There have been 
a few attempts along these lines, but as menticned in the 
previous section, most were hamperel by: 

(1) attempting to directly apply two-level hierarchy 
technigues @ithout carefully considering their 
applicability, 

(2) attempting to generalize techniques which were not 


aven fully understood in a two-level environment. 
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fhe major thrust ot this thesis is to provide insight 


iato and shed additional light on these problems, 


Storaje Hierarchy Systeas 39 


CHAPTER 3. 


FORMALIZATION OF STORAGE HIGRARCHIES AND RELATED RESEARCH 


3.0 Totroductiona 

In this Chapter a tormalization ot the key 
Characteristics of storage naierarchies is presented and 
performance measures are derived, The reported perfcrmance 


of actual systems is reviewed, 


3.1 Major Parameters of a General Stotage Syste 


Table 2 and Figure 1 illustrate the major parameters of 
a storage hierarchy system. These parametets can be grouped 
into four categories: (1) basic technoloyy, (2) 


configuration, (3) aljyorithm, and (4) program behavior. 


3.121 Basic Technology 


The basic technology parameters, cost/byte, C, and 
average access time, T, are pLimarily dependent upon the 
physical properties of tne Storage device technology. At any 
jiven time the state-of-the-art offers only a limited number 


of (c,T) alternatives that the system deSigner can select. 
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*® C¢ cost/byte 


* T average acsess time 


e L number of levels 

* I iaterconnestion of levels 
e S$ size (Capacity) 

* 8B transfer rate (bandwidth) 


e N number of pytes in page (page size) 


* F tetch 
e« P placement 


* kK replacement 


Table 2. 
Major Parameters of a Storage Hierarchy Systen 
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Figure 1. 
Structure of a Storage Hierarchy Systen 
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3.1.2 Configuration 


The system designer does have flexibility in cryganaziny 
these storage devices. By serial and/or parallel structuring 
o£ the components of a given level ot storage device 
t2candlogy, it iS possible to specify, over a wide range or 
values, the size (storaye capacity), S, and the maximua 
transfer rate (data bandwidth), B, cf the system. For 
axample, if a particular technology provides a ktasic device 
with S=s and 3=b, connecting n cf these devices in parallel 
produces a storage level with S=ns and B=nb,. (To some extent 
the m2chanism and cost ot the organizational structure does 
lafluence th2 overall cost/pyte and average access time of a 


laval, this effect is usually minimal for small values of 


a). 


Jn a more global basis, the designer must determine the 


number of levels, L, in the storage systen, the 


ee ee = a 


Latecrconnections ot the levels, I, and the size, N, ofa 


oe ee ee ee ee ee eae eee —— i 


page (the unit of information moved between levels). 


3.1.3 Program Behavior 


The processor, under program control, produces a 


sequential series of reterences to the storage system. Ihnese 


processor references are in the form of logical address 
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2ferences which serv2 to uniquely identity each individual 
unit of stored informatation (@ege, an d’-bit byte) 
independent of its locaticn (i.e., M!, M2, M3, 2.2). The 
time sequence of logical address references, A, 1s called an 
aidcess trace or addrass reference pattera. In general, each 
ugigu2 program and its input data will result ina different 
processor address trace, For purposes of analyzing the 
effectiveness of the storage hierarchy, the address trace is 
the primary characterization of a program that is needed 
(2.J.-, we don't care what the program's purpose is or what 
Language it is written in, etc., we cnly care about its 
aldcess trace). Thus, the address trace describes the 


pcogram's behavior as observed by the storaye hierarchy. 
3.1.4 Algorithno 


There are three basic decision algorithms that must be 
employed by an automatic storage management system, Fetch, 
FP, decides when and which inftorgation should be moved upa 
l2vel (e.g., from M2 to M!),. Elacement, ?, decides wnere 
inforzation should be placed in a level. Removal or 
ceplacement, &, decides when and which informaticn shculd be 


transferred down a level (e.y., from M*' to M@)., 
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A completely general storage hierarchy alyorithn, H, 
must sonsider all the parameters described above: 
d = £(<Technology>, <Contiguraticn>,<Program>,<Alyorithm>) 

H = £(<C,T>, <L, I, S, B, N>, <A>, <F, P, RD) 
Clearly, attempting to optimize a system with sco many 
paramaters is difticult. Fortunately, it is possible to 
ealiminate from concern or at least Simplify certain 


parameters as explained below. 


3.2.1 Configuration 


Consistent with the titie of this thesis, we shall 
consider only hierarchical interconnections of levels as 
1lllustrated in Figure 1, where T'<P2<T3< etc, and NI<N2@<n3< 
atc. fhe rationale for this decision is elaborated in the 


thesis. 


There are thre2 basic strategies for information 
movemant sizes: (1) select a single page size value, N, 
which ais always usad throughout the hierarchy - this 
approach is used on most contemforary automatic multilevel 
storage systems (e.g., Multics), (2) allow an arbitrary 
cange of values for N to be used - this approach is 


peimarcily used on manually managed storage systems, and (3) 
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s2lect L values of N, a specific unit of transfer is used 
between any two levels of the hierarchy - this approach wiil 


b2 pursued and justified in this thesis. 


3.2.2 Program Behavior 


Each logical address can be represented as a bits as 
shown in Figure 2(a). If the paye sizes, N, are chosen to 
b2 powers of 2, the sat of 2**a possible addresses can be 
partitioned into 2**p pages of N=2**n consecutive logical 
addresses each as shown in Figure 2(b). [Note: the notation 
"2**a" means 2 raised to the power aj. Since the information 
movement between storage levels is accomplished by 
transferring pages, w2 can analyze this interlevel novement 
by merely considering the time sequence ot loyical pages 


references, Ap, callei a page trace. 


Since we allow the paye size to be different between 
each level and reguests are only passed down toa eyiven 
lavel if they cannot b2 satisfied by any higher level, each 
level will usually experience a different page trace though 
all are algorithmically derivable from the same address 
tcac2, In fact, if all address references were broadcast to 
all storage levels, the page traces can be determined by a 
Simple mapping from loyical addresses into logical pages: 


paye aadress = integer( logical address/N ) 
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Figure 2. 
Format of Logical Address 
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where N is the page size for that level. 


3.2.3 Algorithm 


Fhe placement decision, P, is usually unconstrained or 
minimally constrainei and, as a result, has fcelatively 


little impact upon performance, 


A demand fetch policy will be used. Assume that at time 
t a request for logical address a (or, equivalently, 
pt=iateger(a/N'!)) arrives at Level M!. At that instant the 
information may currently reside in M1, otherwise it must be 
found in a Lower level. Under demand fetch, if p!' is in M', 
the raference proceeis, the infcrmation is passed back to 
the processor, and no other page sovement occurs i the 
niecacchy. It pt is not in Mi, a request for 
p?=integer (a/N2) is sant from M! to M@. If p2 is in M#, the 
page is transterred to M! and processing continues as 
dascribed above, otherwise a request for p3=inmteger(a/N3) is 
sent from M2 to M3, atc. Note that under the demand fetch 
policy, information is only moved up in the hierarchy when 
and if it is explicitly demanded (i.e., requested) by the 


processor. 


Although demand tetch is only one fossible tetch 
algorithm, it can b2 shown ({63] that for hierarchically 


structured storage systems: 
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“ee. given any trace and replacement alyorithm (not 
necessarily using demand paging) ancther replacement 
algoritnm exist that uses demand paging and causes 
the same or few2ar total number of pages to be 
transferred ..." 
In other words, aS you miyht intuitively suspect, moving 
pages only when necessary results in the minimal number of 
page movements. Of course, if page movement is required and 
the higher level that is to receive the page is already 


tull, the removal aljoritom must be employed to’ provide 


space for the new page. 


3.2.4 xXevised Storage Hierarchy Model 


Based upon the discussion ateve, we can slightly 
simplify the parametacs remaining fcr consideration in the 
storaje hierarchy ailjgorithm, H, so that it need consider 
only: 

H = £ (<Tecnnology>, <lconfiguration>,<Program>, <Algorithm>) 
H = £(<C,I>, <L,S,5,N>, <A>, <K>) 
in this thesis all of these parameters will be considered 
and investiyated, 3pecial emphasis will be placed on 
analyzing and understanding the relationship between the 
pages Sizes, N, and the removai algorithm, &, required for 


atficient operation of the storage hierarchy. 
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3,3 Pectormance Heasures 


There are various performance measures that we could 
consiler. For an overall point of view, system measures, 
such as job throughput, job turn-arcund time, and processor 
utilization, are quit? significant, Unfortunately, it 1s 
extremely difficult to directly relate these measures to the 
performance of the storaye system, even an approximation 
would require consideration of many more parameters. Thus, 
w2 will only consider measures that relate to the effective 


performance of the storage hierarchy. 


3.3.1 Performance Measurement Notation 


Due to the strict hierarchical structure of our storage 
system andthe demani tetch pclicy, we can analyze the 
performance of the system by separately considering tne 
devels of the hierarchy starting with M1. Since a given 
davel only receives a paye fetch request ir the information 
has not been found at a higher level, each level usually 


sees a different page trace, Ap!, Ap@, Ap3, etc. 


There are several important properties of paye traces, 
If P is a particular paye trace (e.y., Ap!) of a proyram, we 
defines 


e |PI length of the page trace sequence 
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° Q set of distinct pages referenced in P 
° 1Q] numper of pages in Q 

For example, in the page trace 
P =a, Dy a, Cy D, a 


w2 observe that 


IP] = 6 
Q = fa, db, 3} 
IQ) = 3 


(Lower case letters will be used to represent loyical fage 


alidresses instead ot page numbers). 


For a specific storage hierarchy, we define jM]| to be 
the size of M in units of pages receivable from the next 


lowec level. For example, |Mtj=S!/N@, {M2,=S2/N43, etc, 


For a specific page trace, P, storage level, 4M, and 
cemoval algoritha, R, we define the result page trace or 
Eatch page trace, P', as the time sequenced page references 
of P that were not found in 4. We shall call page 
references that are found in MM successes. The success 


function, Sf, is the number of references satisfied by M and 
can be computed as 4Pj-{jeP'|t. By analogy to the success 
function, the number of reterences not satisfied by M, |P"]|, 
is called the failure function, Ff. In general, we wish to 


maximize the success Function or, equivalently, minimize the 


failure function. It is convenient to normalize the failure 
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function py defininy the failure frequency function, f, 

f = IP'IsIPi 
The success frequency function, s, can be easily computed as 
1-f£; it is often called the hit rate on a two-level storage 
system. We also define the system failure frequency 
function, £9, of a level to be: 

ro = {Pt Tslal 
where A is the address trace generated by the processor and 
{Aj is the length of tne address trace (it is also true that 
AJ always equals {P?], thus they may be used 
interchangeably). The system success frequency functicn is 


correspondingly detined as s9=1-f°, 


If we apply th2 definitions above to the processor 
generated page trace, P?, received by M!, we note that the 
result page trace, P*, is essentially the page trace, P@, 
received by M2. fhere is a sunoE relabeling reguired to 
adjust for the difference in page size used by #2, 
p2=P'(NI/N2), By repeating this process recursively, we can 
davelop the effective page traces, failure and success 
functions, and failure and success frequency functions for 
@ach level of the hierarchy. Since we assume that all 
referanced information exists in the storaye hierarchy, the 


sum of the system success frequency functious must be 1. 


dne general measure of a storage hierarchy's 
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pacformance is its effective access time, T', and effective 


cost, C', which are defined as fcllovws: 


T° Tis®l+eP2sO2eT3SIF+,,, 
C8 = (C191 +lT2S2+C3sae, op f(S1+S2+S3+...) 

T*' and C* can be view2d as characterizing the entire storage 
hierarchy according to a corresponiing one-level system. 
From a cost/performance point of view, one should be 
iadifferent between a single-level single-technology storage 
device with average access time, T', and average cost/byte, 
cf, and a storaye hierarchy system with performance 
parameters ([f',C"'). In particular, if the system designer 
needs a storage performance (T,C) and no such = basic 


t2chanoloygy exists, h2 must attempt to develop a stcrage 


hierarchy such that (f',C*') = (T,C). 


3.3.2 Page Trace Simulation 


Ine way to determine the success frequency function and 
the result page trace for a specific page trace is to 
Simulate the storage management alyorithms and note the 
contents of M at each step of the page trace. Clearly, 
these results depeni upon numerous parameters (€.Ge, 
specific trace, ramoval algorithm, size of M, etc.). Figure 
3 illustrates this stap by step Simulation assuming demand 
paging, FIFU (first-in first-out) removal, and |Mj| = 2 


pages. For simplicity, the page trace, P, has peen 
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(£3) 
parameters 
* P == 4, Dy by Cy Dy a, dy Cy a, A 
e je) = 10 
= 2 = { a, b, c, d } 
* {21 = 4 
© IM} = 2 


e FIFO Renoval 


Page ITrace,P | a] bj bl] cj bfj{aftdijcfatjfja is 
we eee ee ee $m ta at a tt = te tt 
Fetch t* 4 * 1 1 * | [oe ke Sp ee | 
we ce nn eee Dn ne ee ee eee 
M Contents {aidbifdbjctciyat~dtitciryatdiad <-"*new" 
{after each | j/afdafjbytbiyci4yayatctc ij <-*old" 
reference) | | j | | | | ) | j J 
meee en ee ee $n mba am at na a ta a ta ta a te te ett 
Page frace,P*j aj pb } | c J J}ajdijc fifa} | 
aesults 
e PE = {P"y =7 
© Sf = 4Py-iP"| = 3 
» £ = 70% 
*» s = 30% 
e P*" = a, b, c, a, dy Cy a 


Figure 3. 
Example of Page Trace Simulation 
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uycmalized to be exprassed in units of receivable pages. In 
particular, if M is M!, then J[MJ=S!yN2 and p=integer (a/N2) 
where a is a loyical address reference and —E is 
corresponding page refarence,. the pages in M are shown as 
ordered to indicate the FiFO ordering, the top paye is the 
"last" ("latest") pajye fetched into M, whereas the botton 
page is the "first" ("oldest") page in M and is the page 
selected for replacement when necessary. The asterisk (*) 
indicates that a tetch was reyuired frcem a lower level of 
the hlerarchy, the page reference is thus noted as fart of 


the result paye trace, P*, 


{It is normally assumed that all levels, except level L, 
ace eapty initially, thus there is a transient stage during 
which pages are loaied into M without any replacements 
needel, Since there are so few pages in M during this 
Start-up stage, there are many fetches required. We will 
Find it usetul to separate out this transient phencmenon, 
This transient consists of the page trace up to the tirst 
}4{ unigue page reterances, in the example otf Figure 3 this 
1s the first 2 page references (i.e@., a, b)s Consider the 
case 1f JQYS|M|, there would be no further tetches into this 
lavel after the initial transient that loads the |Q] fages 
into M. In this case, {P'J=1Q] exactly, independent of {Pi, 


and s tends toward 1 as {P| increases. 
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In the particuiac example illustrated in Figure 3, we 
note that there were 3 'hits' and 7 ‘misses out or 10 page 
cafarences, so that s=30%*. Thus, P! cnly consists of 7 paye 


caferences to the lowar levels. 


3.4 Related Research 

As noted above, we wish to develop a storage hierarchy 
with attractive cost/performance, (C*',T"), characteristics, 
It is clear that wa can arbitrarily decrease the cost/byte 
Oy making the size of each level, S, increasingly larger as 
we go trom the high-performance high-cost to the 
low-performance low-cost levels Waddix C1>C2>C3>,.. and 
SIECS2<S3K,..). fn fact, this approach is the basic 


motivation for storaye hierarchies, 


Unfortunately, if the processor generated address 
ceferences that were uniformly distributed in time and 
aidress, each byte would be equally likely to be reterenced 
at any instant. This probability would be: 

Pr[ reterance aj = 1/(S1+524+53+...) 
Thus, the expected system success function, s%®, tor each 
level is proportional to the size of the level. For example, 
S91 = SI/(S'+S24+5S3+...) 6 


uut, since we have assumed that S!<S2<S3<¢.,.., we tind that 
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59'<¢592¢593¢,,, Thus, the system success function for the 
Lth lavel dominates (i.e., 1S approximately 1) since we have 
assumed that it is the largest level. Referriny back to our 
d2finition of effective access time, we find that T*' would 
D2 approximately equal to the lowest performance levei 
(level L) since ali the other terms would be neyliyible. If 
tais analysis were trues, our stcrage hierarchy would result 
un a performance just slightly better than our Lowest 
pertocmance level at a moderate increaSe 1h price - hot an 
especially exciting result. Fortunately, actual storage 
uleracchies do not behave this way. We will briefly review 


some related research on this subject. 


324.1 Locality 


It haS been empirically observed that actual programs 
cluster their references so that, during any interval ot 
tim2, only a subset of the intormation availabie is actually 
used, A detailed discussion of this phenomenon will be 


presented in the thesis, 


It 18 important to note taat due to our basic rankings 
of page sizes and access times in the storage hierarchy, 
2ach level “sees" a different view of the program. ‘he high 
lavels of the hierarchy must fcllow the micrcescopic 


hnstruction by instruction reterence pattern whereas the 
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middle levels follow a more gross subroutine by subroutine 
pattern. The very low levels are primarily concerned about 
the processor's references as it moves from subsystem ta 
subsyster,. #e do not have any a priori guarantee that 
ioicality of referenc2 holds equally true for all of these 
views, but we do nav2 some repcerted evidence to encourage 
us. Most or these studies have been basea upon twe-level 
storage systems or restricted forms of three-level 


aleracchies, 


3.4.2 Paging Systems 


The earliest automatic storage systems were based upon 
two-level core-drum hierarchies (devices 2 and 4 of Table 
1). This technique was introduced in the Atlas system 
i 38,57] duriny the early 1960's. It has since been used on 


many contemporary systeans. 


The performance of paging systems has been Studied by 
varlous researchers, such as Belady [12], Coffman and Varian 
[19,86], Hatfield [48], and Sayre [77]. In Coftman's 
rasults, for example, it was noted that even thouyh 
S$§/(S!+S$2)=09.25, s! otten exceeded 95%. Hatfield studied 
the parformance of system programs that had been carefully 
d2signoed and found that for S!/(Si+S2) ratios as low as 


3.25, it was possible tor s! to crten exceed 99.99%. 
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3.4.3 Cache Systems 


cache systems are based upon two-ievel cache-main 
hieracchies (devices 1 and 2 of Table 1). Although they have 
b2en proposed as early as 1965 jsee Wilkes [88]), the major 
commercial use of cache systems did not occur until the 
introduction of the IBM System/360 Model 6&5 j21,61]. More 
recently, this tecanigue has been used in several 
contemporary systems, such as the ITEM System/370 Model 155 


and Model 165 {52}. 


In these cache systems, IBM found that it was possible 
to drastically reduce 5!/(Si+#S$2) to as low as 1% and still 
c2ep the hit ratio, s!, above 90%. Similar findings were 
ilso ceported by sell and Casasent [13], Mattson { 64], Meade 
“95], and Seligman {78 ]. 


3.4.4 Three-level Systems 


There have bean a few three-level systems reported in 
the literature, unfortunately they have all been saounak ad 
ic) 06uin design and the resuits are far from conclusive. 
There have been at last three types of sucha hierarchies 


studied, 
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3.4.4.1 Main-Bulk-Mass Store Hierarchy 


[There have been several systems devised based upon 
devices 2, 3, and 5 of Table 1, The Buik Store actually 
used, called Large Core Store (LCS), had a much lower access 
time (around 8 us) ani a much higher frice (about 25¢ysbyte). 
In order to compensate for peculiarties in the hardware 
structure and out of considerable concern for the extreme 
cost of LCS, these systems tended to become much more 
manually managed hierarchies than automaticaliy managed. 
Although they were found to be effective, it is difficult to 
g2neralize the results. The most ambitious attempt reported 
was undertaken by Carn2jie-Mellon University [36]. kesults 
have also been reported py Durae [31], Wiiliams ;{389], and 


others. 


3.4.4.2 fain-Larje-Mass Store Hierarchy 


There does not appear to be any autonatically managed 
systems of this type published in the general literature, 
The Multics system at MIT Froject MAC has recently 
iotcoduced a “page-multilevel" strategy based upon devices 
2, 4, and 5 of Taple 1. There has only been Limited finding 
rapocted to date but it has been stated in the March 1972 


issu2 of the MIT Information Processing Services Bulletin 


a 
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(p. 17) that it 


"... does pay off since it meets fluctuating demands 

on the system, reduces the werkload for the disks to 

an efficient level, is inexpensive, and keeps pages 

on the drum for an acceptable length of time." 
AS an indication or its effect, the new strategy is reputed 
to have increased the success frequency tunction, s?, of the 
dcam from 29% to more than 90% (i-e., “reduced from one page 
cead from the disk for every four reads from the drum, to 


one page read from the disk for every ten to twenty fages 


from the drum"), 


3.4.4.3 Main-Larye-Giant Stcre Hierarchy 


The work of Considine and Weis [20] is difficult to 
categorize. It is based upon a three-level hierarchy where 
the first level corresponds to device 2 (main store) of 
Table 1, the second level corresponds to a combination of 
devices 4 (drums) ani 5 (disks), and the third level 
consists of removable disks which can best be approximated 
oy device 6 ot Tabl2 1. It is impossible tc compute any 
success freyuency ftunctions from their data, but it apfpears 
that for 352/(S2+#53)=0.5, s? ais very high. They note 
(p.44)), in particular, "most of the data moved to the 


archival storage (i.e., 43) have stayed there," 


3.4.5 Need for 


Althoughi 
gacouraging, 
aialtiple-level 
This thesis is 


area, 
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Additional kKesearch 


the results of research described above is 
the design and performance ot general 
storage hierarchies are still inconclusive, 


intendel to provide specific results in this 
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CHAPTER 4, 


A STORAGE HIERARCHY SYSTEM 


4.9 Introduction 

In this chapter a design for a general multiple level 
storaje hierarchy system, in particular with tnree or more 
levels, is presented. This design is based upon an crderly 
and uniform treatment of the logical structure of the 
stocaje levels and their interccnnections, In addition to 
providing a solution to convenient stcrage manayement for 
the user, this design is intended to produce good 
pertormance for the storage hierarchy as measured by its 
atfective access time, T', and effective cost, C'. The 
poinciple and novel techniques to be used are described 


separately in the sections below. 


2S a Se eS ee eee 


4s noted earlier, automatic storage hierarchy systems 
ace still in the minority. Amongst those systems that do 
provile automatic storage hierarchy management, the majority 
limit their scope to two levels with a few care three level 


systeas, As a result of these limitations, the user is 
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still forced to rely on manual or semi-automatic storage 
wanagement techniygues to deal with the storaye levels that 
are not automatically managed. Thus, an automatic storaye 
management system should consist of a continuous hierarchy 


that ancompasses the full range of storage levels, 


4.1.1 Cost/Performance of Adjacent Levels 


A major obstacle to generalizing storage manayement 
algorithms, in partisular in two-level payiny systems, 1s 
the tremenious contrast, often over 3 crders of magnitude, 
in cost/performance between M! and M2, As illustrated in 
Table 1 (page 28), a representative Main Store, M!, has an 
access time of 1.44 us compared to a Large Store, M2, with 
an access time of 5 ms. In such a two-level system, the 
affective access time, f', is 

~* = TtsO1 ¢ p2so2 


™ 1.4459! + 5000892 


and since s®lts®2=1, wa can substitute s®l=1-s92 to get 

T' = 1.44 - 1.44592 + 5000s92 

Tt = 1.44 + 4996,56592 
In orier to attain an effective access time, T', that is 
comparable to the Main Store access time, T!, we must keep 
tae system success rr2quency tunction, s%@, very close to 9 
Or, correspondingly, xeep s®%! very clcse to 1. Even with 


s92 at 99.8%, an improvetent to 99.94 would cut the 
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atfective access time, T*, in half. With such pressure to 
attain very high s°! values, the systems designer is often 
roarcei to seek out vary specialized techniques in contrast 


to our goals of orderly and uniform algorithms. 
4.1.2 Moderate Cost/Perrormance Ratics 


In order to make the storage hierarchy design robust 
and flexible, the cost/performance characteristics should 
diftec by less than two orders of magnitude between adjacent 
levels, Thus, success trequency functions in the range 90% 
to 399A are adeguate t3 insure reasonable performance, If 
the differences are much greater, it wili be difficult to 
tini sufficiently efficient yeneral algorithms. Since minor 
changes in production techniques and technology evciution 
can result in a variation of a factor of two or three in the 
cost/performance for a yiven technology, it is not desirable 
to decrease much velow one order of magnitude difterence 


between adjacent storage levels, 
+.2 Shadow Storage ani Page Splitting 


The time, Ta, required tc move a page between two 
davels of the hierarchy usually consists of summing two 
components: (1) the average access time, T, and (2) the 


transter time, BxN. 
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If all page sizes were set to provide exactly the 
amount of information, N!, requested by the processor, the 
page movement time would be 

Tm = T + BxN! 
where T and 8 woull depend upon the particular storage 
devels. By examininjy the representative uevices shown in 
Table 1 (page 28), we see that access time varies much more 
than transter rate (i.e., access time spans 6 arders of 
magnitude whereas transfer rate varies by only 3 orders of 


magnitude). 


4.2.1 Marginal Increase in Page Transfer Time and Reference 


Probability 


Let us assume that N? iS quite small, such as a bytes. 
W2 can ask the question: What is the marginal increase in Tu 
it we transter the adjacent N* bytes in addition to tne N!? 
bytes reyuested by thea processor? fable 3 on the next page 
answecs this guestion, Notice that the marginal increase in 
fa decreases from a high of 5.5% (level 2 to level 1) toa 
low of .0024 (level 6 to level 5). This fact iS only 
lateresting if we also consider the concept of locality (see 
chapters 5 and oo for additional discussion) and the 


question: What is the probability, Fr, that the processor 
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(ft 3) 

Laval Lia Tm Marginal Increase 
feansfer (7 unit) (2 units) in Tm 

2 to 1 (*) 1.44 us 1.52 us 5.5% 

3 to 2 137 us 132 us «8% 

4 to 3 5906 us 5011 us 21h 

5 to 4 38010 us 38020 us ~ 03% 

6 to 5 600013 us 600027 us »002k a 

Table 3, 


Marginal Increase in Page Transfer Times 


* The figures for access time and transfer rate for the 
Main Store listed in fable 1 are approximations that are 
only neaningtul for very large page sizes. For the page 
sizes under consideration in this chapter, the figures used 
in tha table above are more apprcpriate. 
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Will ceference the adjacent N! bytes with a snacrt interval 
of time, such as Tm seconds? Due te locality of program 
teference, we would expect Pr to be much larger than mereiy 
the reciprocal of the logical address space size, 
Furthermore, Pr shouli increase as Tm increases. Thus, for 
a given level, if Pr is larger than the marginal increase in 
Ta, it is beneficial to transfer the additional N! bytes ana 
thereby avoid the necessity of expending Tm seconds to 


transfer these N! bytes later separately. 


These Same arguments can be applied to the guesticn of 
transferring the adjacent nxN?} bytes, etc. Since the 
arginal increase in Tn decreaseS monotonically as a 
function or storaye lavel, the number of N?! byte packets to 
be transferred as a Single page should increase 
monotonically. This confirms cur earlier decision that 
NI<CN2CN3C etc, 


> 


4.2.2 Choice of Paye Size 


In order to simpliry the implementation of the systen 
and t23 be consistent with the mapping from logical address 
to page address illustrated in Figure 2 (paye 46), we will 
reguice that all page sizes be a power of two. Thus, each 
page size (e.g., N43) 1s some pcwer of two larger than the 


page size of the aext hiyher level (@e.ge-, NI=N2¥**1), 
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Cleatly, the speciiic values of Pr and thus the choice for 
aach page size depenis upon the characteristics of the 
programs to be run and tae eftectiveness of the cverall 
storaje systea, Preliminary measurements indicate that a 
ratio of 4:1 between levels is reasonable, Meade [65] has 
creported similar findings. Other important factors 


atfecting paye size are discussed in Chapters 5 and 6. 


4.2.3 Page Splitting 


Now let us consiljer the actual movement of information 
ia the storage hierarchy. At time t, tne processor 
jyanerates a reference tor logical address a, Assume that 
tne corresponding information is not currently stored in M! 
or M2 but is found in M3, For simplicity, assume that paye 
sizes are doubled as we go down the hierarchy (e.g., N@=2N?, 
N3=2N2=4N1, etc.; s22 Fiyure 4). The page of size N3 
containing a 1S copiad trom M3 to M@, M2 now contains the 
needed information, sd we repeat the process. The page of 
size N2 containing a is copied from M2 to Mt, Now, finally, 
the paye of size N42 containing a is copied trom M! and 
tocwarded to the processor, In this process the page of 
information is split (i.e., page splitting) repeatedly as it 


gaoves up the hierarchy. 


Mi 


“3 
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(£14) 


Processor 


Figure 4, 
Page Splitting and Shadow Storaye 
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4.2.4 Shadow Storage 


As a result of this splitting, the page of size Nt that 
is ceceived by the processor has lett a "shadow" consisting 
of itself and its adjacent payes behind in all the iower 
levels (i.e., Shadow storage). Presumably, if the program 
exhibits locality ot reference, many of these shadow fages 
will ba referenced shortly afterward and be moved further up 


1n the hierarcay also. 


4.2.5 Copying ot Pages 


in the strateyy presented, fages are actually ccpied as 
they move up the hierarchy; a page at level n has one copy 
of itself in each of the lower levels. Since processor 
"fetch" reyuests substantially outnumber “store"™ requests 
2.j-, by more tnan 5:1 ain Some measured programs), the 
contents of payes are seldom changed. Thus, 1f a page has 
not b2en chanyed and is selected to be removed frem one 
level to a lower level, it need not be actually transferred 
since a valid copy already exists in the lower level. The 
contants of any level of the hierarchy is always a subset of 
the information contained in the next lower level. Thus, 
tae total intormation capacity of the system is eyual to the 
size of the level iL store rather than the sun cf the 


capacities of all the lavels. Since the capacity of level L 
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is assumed to be much larger than the capacity ot L-1, etc., 
the difference in total system capacity due to shadow 


storage is minimal, 


In the description above it is implied that information 
actually moves between adjacent levels. This approach, 
callei direct transfer, is indeed intended. sy Ccmparison, 
taough, many proposed and experimental multiple level 
storaje systems are based upon an indirect transfer (Cede, 
the Multics "page multilevel" system mentioned in Chapter 
2). [In these systems, all infcrmation 1s routed through 
level 1. For example, to move a page from level n to level 
n-1, the page is movei from level n te level 1 and then rrom 
ievel 1 to level n-i. Clearly, this indirect approach 1s 
and2sirable since it requires extra page movement and 
consumes a portion of the limited M! capacity in the 


process, 


There nave been two major obstacles to direct transfer 
in previous systems: (1) interccnnection structure and (2) 


Synchronization. 
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4, 3.1 Interconnection Structure 


For many reasons, some technical and some historical, 
most contemporary systems are physically structured in a 
cadial manner, That is, there is a central element to the 
systena, either the processor itself or the frimary store, 
and all other storage devices and/or processors are directly 
connected to this central element, Except for some possible 
control sigaals, th2re are no direct data transfer 
connections between the non-central elenents. This 
structure is, of course, quite consistent with a 
hon-aierarcnical storaye management systen. A logical 
storcajye hierarchy system should be based upon a physically 


nleracchical interconnection structure. 


4, 3.2 Synchronization 


AS indicated in Table 1, storage devices often have 
diurrerent timing and transfer rate characteristics. In crder 
to accomplis a direct data transfer between levels, 
Synchronization is necessary. It may be oavious that a 
stocage device can not transfer data faster than its rated 
pecformance, but for many storage devices, especially 
eleactromechanical devices, it is not possible to transfer 


data slower than its cated speed. 
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Based on current technology, this problem can be 
solvel, Many of the storage devices ace now 
non-electromechanical (i.e., strictly electricai), suca as 
tne Cache, Main, and Bulk Stores of Table 1. it is yguite 
f2asinle to provide jirect transfer between any ot these 
devices and any other storage device; this is one reason tor 
the cadial interconnections described above where the Main 
store acted as the common means of provading 
synchronization. Using a Similar approach, we can allow 
direct transfer between electromechanical devices if this 
trcanster is routed through a small and reasonably 
inexpensive electrical storage buffer. Femling L33] 
discusses such a devise, which he calls a cubber-bana memory 
peesumably because it "stretches" to watch the 


characteristics of tna source and destination devices. 


In the description above, it is implied that a transfer 
up the hierarchy trom level 2 to the processor (level Q) 
consists of two sequeatial steps: (1) transfer paye ot size 
N2@ trom level 2 to level 1, and then (2) extract the 
appctopriate page subset of size N! and transfer it fro 
leval 1 to the processor (level 0). In general, a transter 


from level n to the processor would consist ot a series of na 
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steps. Thus the system page transfer time wculd equal the 


sun of n inter-level page transfer times (@.ge, 
Tai®+T m2seTmse+ eoode Furthermore, for nany 
e@lectromechanical storage devices, the second access, 


required to forward the page subset, may experience the 
“nakiaum" access delay rather than the "average" (i1.¢e., 
after storing the information into the level, a complete 
m@Cchanical revolution may be required to reposition to read 


the same information and forward it to the next level). 


Phis inefficiency can te avoided by allowing 
information to ba stored into all upper levels 
simultaneously. Figure 5 illustrates this mechanisa, If 
information is to be transferred from M3 to the processor, 
43 turns on its output data gate, G3out, when it is ready to 
start and transfers N3 bytes and their correspending legical 
addresses up the data bus. M2 turns on its input data gate, 
3s2in, to receive these N3 bytes; furthermore, when the 
appropriate N2@ bytes neeled py M! are detected by &2#, it 
turns on its output data gate, G2out, and these N2@ bytes are 


forwarded to 4! while being stored in M2, etc, 


For example, assume a reference tc lcyical address a is 
yenercated by the processor and the corresponding information 
i35 current stored at level na (and all lower levels, of 


course). At the instant that the Ni bytes containing a are 
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Figure 5, 


Read Through Structure 
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placed on the data bus by level n, these N! bytes will be 
stored into all lev2ls from level na-1 to level Q (the 
processor) Siinultaneously. Likewise, the N2 bytes 
containing a are Simultaneously stored into all levels fron 
level n-1 to level 1. This strategy thus makes it appear 
that the N! byte page requested by the processor is read 
through directly to the processor without any delays. 


4.4.1 Page Transfer Time 


Jsing the read throuyh strategy, the page transfer time 
to the processor is actually less than the page transfer 
time to the adjacent storage level. For example, if the 
raguested information is stored in M3, the page transfer 
time to the processor, via read through, is 

fia3° = TRS + Nilp3 
whereas, the page transfer time from M3 to M4 is 
Im32 = [3 + N3B3, 


Since Ni<N3, then Tm3°CTm32, 


4.4,2 Availability ani Servicability 


The read through mechanism descriked above otters some 
Luportant advantages to the availability anu serviceability 
of th2 storage system, Note that all storage levels are 


connected to the gatei data bus not directly to each other. 
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I€ a storage level aust be removed from the system for 
servicing, it is merely necessary tc manually set both Gin 
aad Gout "on", In this case, the intormation is' really 
“cead through" this Level as if it didn't exist. No other 
Caanges are needed to any of the other storage levels or the 
storaje management algorithms although we would expect the 


performance to decrease, 


425 Store Behind 

Undec normal steaiy-state operation, all the levels of 
the storage hierarchy wiil be full (except fossibly level 
L). Thus, whenever a paye is tc be moved into a level, it 
1S mecsessary to remove a current page. If the page selected 
foc cemoval has not been changed by means of a processor 
"store", the new page can be immediately stored into the 
lav2l since a copy of the removed page already exists in the 
n2xt lower level of the aierarchy. If the processor 
j2znerates a "store" reaguest, all levels that contain a copy 
of tha information being wodified must be updated. This can 
y2 accomplished in three basic ways: (1) store through, (2) 


stoc2? replacement, or (3) store behind. 
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4.5.1 Store Through 


Under a store through policy, all levels are 
Simultaneously updatel whenever the pHOGessox generates a 
"store" reyuest. This is the obvious inverse of the read 
torough policy. But, there is a crucial distinction. Under 
reai through, only storage levels 1 through a are used, 
wher2 n is the highast level containing the requested 
information, Store through must update the contents of 
levels n through L. [faus, rcead through speed is limited by 
its slowest level arfacted, level n; store through is always 
limited by the speed on level L, the slowest level of then 
all. If 20% ot all processor requests are "stores", the 
syste2m success freygu2zncy Function ct level L wiil be at 
least 20%. Due to its large average access time, level L 
will be the dominate portion of the system's effective 


access time, T'. 


Store through can be used efticiently only if the 
access time of level L is comparable to the access time of 
l2vel 1, such as in a two-level cache systen, In fact, it 
is used in some cache systems, such as the IBM System/370 


Mod2ls 155 and 165 [52]. 
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%.5.2 Store Replacement 


Under a store Ceplacement policy, the processor only 
stores into M!,. Ir a chanyed page is later selected tor 
c2moval, it is then moved to the next lower level, M2, 
Lamediately prior to being replaced. This process occurs at 
avery level and, eveatually, level L will be updated but 
only after the paye has been selected for removal trom ali 
the higher levels. Due to the extra délays caused by 
apdating changed payes pefore teplacement, the effective 
acca2ss time for fetches is increased. Various versions of 
Store replacement are used in emecst two-level paging systems 
since it offers substantially better performance than store 


through for slow second level storage devices (e.3., drums 


and disks), 


4.5.3 Store Behind 


Store Behind is a compromise strategy that bridges the 
jap between store through and store replacement and offers 
substantially better performance, In both strategies above, 
the storage system was required tc perform the update 
Operation at some spezific time (e.g., at the instant of the 
"store* reguest for store through or at the instant of 


cemovwal for store replacement). Once the information to be 


storej has been accepted by the storage management systen, 
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the processor doesn't really care how or when the copies in 
the storage hierarchy are updated, Store behind takes 
advantaye of this dajree of freedon, Due to the Large 
jisparity between average access time and transfer rate for 
most levels, the maximum data transfer capacity is rareiy 
reached (i.e., at any instant of time, a storage level may 
udt have any outstanding requests for service or it may be 
waiting for proper positioning to service a pending 
request). During trese "idle" periods, data can be 
transferred down to the next level of the storage hierarchy 
without affecting or delaying any fetch operation. Since 
these "idle" periods are usually very frequent under most 
actual circumstances, thera can be a ocontinual flow of 


caaajyed information down through the hierarchy towards level 


Although an effective storage management system should 
attempt to minimize page movement and its associated 
"housekeeping", there will still be a substantial amcunt of 
WOCK Leyuired to tmanaye the hierarchy. It is desirable to 
camov2 aS much as possible of the storage manayement [rom 
tne concern of the processor and the programs running on the 
processor, including the operating system. There are two 


peigacy motivations for this cbjective: (1) the stcrage 
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hierarchy should function as an independent component of the 
system to eliminat2 any added complexity to the processor or 
projrams, and (2) w2 want to conserve the processor's 
computational powers for solving the user's problems ratner 
tnan for "system overhead". In actuality, of course, the 
storage hierarchy can not be divorced entirely from the rest 
o£ the system, but th2 remaining interdependencies should be 


minimal. 


4.6.1 Distributed Control 


In the hierarchical storage system described above, all 
storage management operations can be determined local toa 
Single level or, at most, in consideration of information 
trom aeighboriny levels. Thus, it is possible to distribute 
toe control of the nierarchy into the levels, this also 
facilitates parallel and asynchroncus operation in the 


hierarchy. 


In a comprehensive multiple level storage hileraichy, as 
illustrated in Tabla 1, this automatic and distributed 
control can be accomplisned by using two mechanisms: (1) 


processor functions, ani (2) "intelliyent" controllers, 
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4.6.1.1 Processor Functions 


The management of the tirst storage level must cperate 
at speeds comparabie to the processor. AS a result, it 1s 
usually necessary to incorporate the first level store and 
its associated manajement operations into the processor 
nardware itself. This approach is used in the iBM 


System/370 cache systeas [ 52]. 


It is often desirable to incorporate the management of 
the second storaye level also into the processor, This 
l2vel requires substantial performance to nandle the demands 
for service from the first storage level. Since its 
c2yuicements are not guite as demanding as the first level, 
it is an ideal candidate for tfirmware control, assuming that 
tae processor iS mMicroprogrammed, This approach has not been 
useid ain any current commercial systems, although the 
iatejyrated (i.@., microprogrammed) channels of certain 
wodels of the IsbM System/370 are based upon similiar 
concepts. fhere have been a few experimental systems, such 
as the VENUS System at MITRE, whick provides processor 
functions to essentually manage the paying system via 


miCroprograaming. 
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4.6.1.2 "Intelligent" Controllers 


Por the third storage level and beyond, the storaye 
manayement performance requirements are much more’ wodest 
since most of the storage activity should occur at the first 
and s2cond levels. For these lower levels, it is possibile 
to develop indepenient storage management ccntrol facilities 
for each level, This can be accomplished by extending the 
cunctionality of conventional device controllers. Some 
crecant sophisticated lavice controllers are microprogrammed 
and ace already capable of perfcrming the storage manayement 


function [1]. 


4.6.2 Multiprogramaming 


Jp to now we have tacitly assumed that the prccessor 
bacom2s idle whenever it is necessary to fetch information 
trom the storage hierarchy. This may be a reasonable policy 
for two-level cache systems since the processor 1s never 
idle for more than one or two microseconds at a time. But, 
for paying systems and general multiple level storage 
hierarchies, the processor may be idled for periods of 
hundreds or thousands of microseconds at a time. It 1s 
worthwhile to try to rind useful work for the processor 
While the storage hierarchy is retrieving the requested 


information, 
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In most conventional computer systems, processor idle 
time is utilized by multiprogramming. This requires that 
taere be multiple programs available to be run. Whenever 
one program must be delayed due to a time-consuming storage 
caquest, the processor is switched to another proyran, 
Under reasonable circumstances (@€.6g., many programs ready 
for axecution and moderate load on the storage system), it 
is possible to keep the processcr ccntinually busy. Thus, 
tne affective system storage access time, f', will very 


closely approximate f!, 


Unfortunately, the process cf switching executicn from 
one program to anoth2r can result in a considerable amount 
of processor overhead, For example, an early version of tne 
Multics operatiag system was reported to require 19 
milliseconds to switcn programs; typical operating systems 
reguire up to 1 millisecond. The time required to 
accomplish this multiproygram switch can be drastically 
reduced if the multiprogramming management is also 
incorporated into th2 processor along with the rirst and 
seconi storaye level manayement. Although the particular 
purposes were ditferent, hardware supported multiprogramming 
aas been available on several computing systems, such as the 


Higney@ell 8399 series [46] and more recently in tne Singer 
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System Ten [30]. The less frequently executed operating 
system functions, such aS job scheduling and time-sharing 
management alyocithws, can be supported by the software 
operating system as on conventioal systems without adversely 


atfecting performance, 


4.7 Comments gon the Storage Hierarchy System Design 


This chapter has presented the key concepts of a 
general multiple level storaye hierarchy systen. Many of 
the particular details or the systen will require 
consiljerable investigation and experimentation to determine 
an optimal implementation. Three important factors are 
axtensively studied in the followiny chapters: (1) other 
page size considerations, (2) removal alyorithms, ana (3) 


crelevant models for proyram r3ference behavior. 
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CHAPTER 5, 


ANALYSIS OF PAGE SIZE CONSIDERATIONS 


5.0 Introduction 

Ine of the most important parameters or a _ storage 
nieracchy system is the page size, the unit of information 
tcanster between two levels of the hierarchy. In this 
chapter, the factors influencing page size are examined from 
the device characteristics viewpoint and the program 


behavior viewpoint, 


Ju contemporary two-levei paging systems (based upon 
two davices similar to devices 2 and 4 of Table 1), the page 
Ssiz2 1s usually quite large (typically 4096 bytes for paging 
systems) to take advantage of M2's large transter rate toa 
compensate for its slow access time. Such a large paye size 
is justified by reliance on the Principle of Locality. 
~onsidering the devices of fable 1 for example, a single 
byt2 can be accessed and transferred between M4! and M2 in 
about 5 milliseconds whereas 4096 contiguous bytes can be 


fatchad in 7.8 millisaconds, only 56% more time, 
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5.1.1 Page Size Investigations 


Although paginy systems have been used successfully, 
tae affect of page size nas become the subject of increasing 
investigation. This interest has been aroused due to several 
considerations: 

1. it has been noted by Denning [26] that the 
utilization of M! is maximized and “page breakage" minimized 
by using rather small pages, such as 200 bytes. In 
particular, ne emphasizes: 

"These results are significant ... small pages 
permit a great deal of compression without loss of 
efficiency. Small page sizes will yiela significant 
improvements in storage utilization ..." 

26 The success of cache systems indicates taat the 


Principle of Locality applies on the micrcscopic scale as 


w2ll as the macroscopic scale cf conventional paying 
systems, 
3. The recent introduction of several new device 


technologies, such as the "Semiccnductor drum" [35] with an 
average access time ot about 160 microseconds, drastically 
ceduces the benefits of very large page sizes in a paging 
systen,. 

4. Although most current multilevel systems employ 


only two lavels, this thesis is ccncerned with muitiple 
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lavel storaye hierarchies (1.e., three cf more levels). In 
fact, storage systems with six or more ievels are yquite 
plausible. A deep unierstanding ot the effects of various 


page sizes is essential to the development of such systems. 


Thus, although there are many reasons’ fcr considering 
new page Sizes, there is not a complete understanding ot the 
impact of such a change. Denning (26] sums up our current 
knowledge as follows: 
"Two factors primarily influence the choice of page 
sizes fragmentation and etticiency of page-transport 
operation." 

In this chapter some other tactors of potentially crucial 


importance will be discussed, 


5e2 Anomalies 

Jne of the more intriguing and frustrating aspects of 
complex systems, such as paging systems, is the occurrence 
of anomalies (i.e., phenomena that are contrary to "ccmmon 
sense"). For example, Belady [10] has shown that certain 
storaje management reaaoval algorithms, in particuiar FIFO 
(first-in first-out), may actually cause performance to 
decrease as the capacity of M! is increased. This result is 
contrary to the general belief that “more main memory saxes 


things work out batter", Thus, one must exercise 


Storage Hierarchy Systems 39 


considerable care when considering "tinkering" with the 
parameters, such aS page size, of a multilevel stcraye 


system. 


The objective of this Chapter is to present and analyze 
some anomalies encountered when the page size parameter is 


chang2d in a paying systen,. 


5.3 The Page Size Anomaly 


Por Simplicity, let us start by consiaerinyg the effect 
of decreasing the paje size used in a two-level system, 5S, 
from N to N®* where N* = N/2 in this new system, S‘'. In 
particular, we wish to investigate the effects upcn the 
failure frequencies which are ft and f', respectively. we 
define the ratio f'/f to be cr. The possible results can be 


partitioned into three interesting regions: 


5.321 Case 1: r <1 ( r' <f£). 

This would be a highly desirable resuit since the 
QNumbec of page fetches is actually decreased. Furthermore, 
the time required to access and transfer a page of size N! 


would be expected to be less than that required fcr the 
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larger page size N. Figure 6 illustrates an instance of this 
case. In converting au address trace to a page trace for N°, 
the logical page addrasses pt and p- are used to represent 
the two halves of the page p of size N. Note that when using 
a page Size of N/2 instead of N, M! actually holds twice as 


many payes though eaca paye is only half as larye. 


In the example of Figure 6, cr = 0.5, which means that 
the number of page fetches was cut in half by using the 
smallar page size N*', This type of result might be expected 
from a program that exhibited a cvather Sparse and 
aon-localized reference behavior. Recall that in typical 
two-l2vel paging systems, a page of size 4096 bytes is 
fetched even thouyh a Single reference uses only a few 
bytes. Unless the program immediately makes many more 
ceferances to this page, much of it will have been fetched 
put not used. Under these circumstances, M! might be better 
utilized by aolding a Larger and more diversified collection 


of pages, even if each paye were smaller. 


5.3.2 Case 2: 1s rs 2 ( t s f£' $ zf ) 

This is a transitional region. For r= 1, S* wiil 
perform better than S since the number of page fetches is 
the same and the time required for each fetch 1S less. For Cc 
= 2, S* wili reyuire twice as many page fetches. This will 


usually swamp any paye transfer benefit derived from the 
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smallar page size, thus S would pertorm better. The specific 
point of transition, ct, depends largely upon the time 
r32quiced to access and transfer a page, T and fT 


craspectively in S and S*', such That r' = T/T. 


Figure 7 illustrates an extreme example of Case 2 wnere 
c= 2.0. This means that the number of page fetches was 
doubled by using tae smaller page size N*. This type of 
c2salt might be expected from a program that exhibited a 


danse, localized, and saguential reference behavior. 


Intuitively, the r= 2.9 result is the “worst™" case 
sinc2 we ar2 being forced to always load both the p+ and p- 
aalves of each original page p, thereby losing ali the 
panefits of the smaller N* page size and incurring twice as 
many actual page faults. This intuitive observation is 


false; r = 2.0 iS not the "worst" case, 


9.3.3 Case 3:3 r> 2 ( £* > 2f ) 

This third cajion, besides being intuitively 
inpossible, is clearly undesirable. Since the number ot page 
tetchas required would be more than doubled, the pertormance 
of S* would be undoubtedly worse than S. Depending upon the 
actual value otf or, the pertormance could be much worse, 
Figure 8 illustrates a reference pattern that produces a 


result of r= 2.75. This reyion of operation will be the 
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subject of discussion tor the remainder or this Chapter. we 


formalize this situation by the following existence theoren. 


(th 1) 
THEOREM 1: 


There exists a page trace, P, and demand-fetch 
PIFO-cemoval two-level storage systems, S and S*, with 
page sizes N ani N'=N/2, respectively, such that the 
catio, r, of tetch frequency f' to f exceeds 2. 

Proof: 

By example (Figure 3). 


see we ew ee we ew ee ee ee eee ee ew ee we ee eee ee ee ee ee ee ee ww we wwe ee 


3.3.4 Other Removal Alyorithms 


Theorem 1 states the anomaly that decreasiny page size 
by a factor ot two can cause the page fetch tfreyuency to 
lacrease by more than a ractor of two. The two-ievel 
demani-fetch conditions of Theorem 1 are typical cf most 
contemporary paginy systeus, But, to put this situatica into 
perspective, other removal algoritams must be considered. 
Due t> its simplicity, the FIFO removal alyorithm was used 
in many or the early paging systems. In recent tiwes it has 
been found tnat FiF) has certain disturbing pecularities 
(e.g., the system's success frequency, s, 1S not a monotonic 
function of primary store size, |M1{ ([10J). Furthermore, 


other removal algorithas have been fcund to be empirically 
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closer approximations to the “optimal" removal alyorithn, 
MIN [11]. MIN itself is not physically realizable since it 
requires future knowledge, but it can be used as a basis for 


parforcmance comparison with practical algorithms. 


Various forms of the "least recently used" (LRU) 
Cemoval alyorithm have become popular in contemporary 
3systeus. Under LRU, the page selected tor removal from the 
primacy store is the one that has not been referenced for 
the longest time (i.e@., the least recently used page). 
Sapicically, LRU has been found to closely approximate the 
pertorcmance ot the “"optimai"® algorithm for many actual 
projcams. Furthermore, Mattson et ail [63] have studied LRU 
and found that it is a member of a genecal class of removal 
algoritnms called "stack algorithms", The class of stack 
aigorithms, as noted by Denning {25], “contains all the 
‘ceasonable' algorithas", In particular, stack algcrithas 
all satisfy an inclusion property that results in well 
behaved characteristics. For example, it has been froven 
that all stack algorithms, including LRU, have a_ success 
treyguency that is a monotonic function cf primary store size 
and immune to the FIFO peculariarity observed by HBelady. 
Thus, one might be tampted to assume that the page size 
anomaly is also a phenomenom unique to FiFO removal and 
would not occur if a “well behaved" removal algorithm, such 


a3 LRU, were used. This expectation can be rapidly destroyed 
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by observing Figure 9, wnich is the same system as Figure 3 
put with an LRU removal algorithm, in this example, the page 
fetch frequency ratio, r, iS 2.2 which still exceeds 2. This 
result leads us to Theorem 2 and Corollary 2a. 
THEOREM 23 
There exists a page trace, P, and demand-fetch 
L&U-removal two-level storage systems, S and S', wath 
page sizes N and N'=N/2, respectively, such that the 
Catio, r, of fetch frequency f' to f exceeds 2, 
Proof: 


By example (Figure 9). 


CIROLLARY 2a: 
3iven a page trace, P, and demand-retch two-level 
storage systems, S and S*', with page sizes N and 
N'=N/2, respectively, the use of a "stack" rewoval 
algoritha (i.e., an algorithm with the “inclusion 
property") ais not sutticient to guarantee that the 


catio, c, of tetch frequency f* to £ will be bounded by 
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ee ee ee ee eee oe 


[The previous theorems prove that there exist paye 
trac2s that result ia significantly increased paye tetch 
frequencies if the payee size is decreased, It is necessary 
toa consider the likelihood of encountering such page trace 
patterns in actual proyrams. For example, it can be proven 
that, aS you are readiny this sentence, ali the molecules of 
air in the room may Suddenly move towards the opposite 
cotner and cause you to suffocate. If you survived the last 
senteaice, you have probably deduced that the likelinocd ot 


that 2vent is extremely suwall, fortunately. 


5.4.1 Simulation Studies 


Hatfield [43] and Seliggan [78] have pertormed 
experiments that indicate that the page size anomaly is very 
common, it not inevitable, in actual programs. In both cases 
actual programs were monitored and their corresgpondiny page 
tcace reference strings were recorded, usuaily on magnetic 
tape. Then simulators were developed that mimicxea the 
software and hardware of the two-level storaye systems then 
in us2 or being consiiered. By supplying the monitored page 
traces aS inputs to the simulators, the performance ct such 
a system Can be accurately measured. These simulators were 


scrupulously accurate, not just approximations. The validity 


Storage Hierarchy Systems 100 


of these results have been confirmed in some cases vy 


cinniay the real projrams under a real two-level storage 


5.4.2 Hatfield Studies 


Hatfield [| 48] performed studies in the hardware 
environment of the IBM System/360 Model 67 with programs 
tanning under the CP-67/CMS Operating System. The simulated 
pertormance was measured for various page sizes, N, and 
various primary store Sizes, |M!]. In summary, it was 
contircmed tunat certain programs, which were viewed as 
axamples of low-density storaye use, resulted in decreased 
page tetch frequency when page size was decreased. But, it 
was observed that for programs with much greater 
localization of heavily used storage?: 

"not only does tha smaller paye size often generate 
nearly twice as many page fetches as the large page 
size, it often resulted in more than twice the page 
tatch2s, contrary to our intuiticns." 
In particular, the substantially increased page fetch 
trequency appears to be: 
"a characteristic of proyrams which have a high 
locality and therefore perform well on systems uSing 
cralocation hardware for address translaticn and is 


characteristic of those programs in the region of 
low paging rate," 
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In other words, the anomaly is most prevalent in programs 
"“aptimized" for performance in a two-level storaye systen 


when running under nearly "optimal" conditions! 
3.4.3 Seligman Studies 


Whereas Hatfield was concerned with a paging system 
with page sizes inthe range from 2048 to 160384 bytes, 
S2ligman ,{ 78] analyzed a proposed cache system with much 
smaller page sizes in the range of 8 to 256 bytes. de 
observed that: 

"interestingly, the missing page probability (for 
this data) is minimized for a page size which 
iacreases slowly with total memory size. Note that 
the associative m2mory organization, where page size 
@qguals one word, is not optimum; tc borrow a phrase 
from economics, the marginal utility of the extra 
words fetched in a page is higher than that of those 
displaced", 


Thus, continual decreasing of page size appears to have an 


inevitable adverse eff2act upon system performance, 
524.4 Other Questions Raised 


Now that it aas been shown that the page size anomaly 
is th2oretically possible and likely to occur in practice, 
there are several other yguestions of interest. Since it has 


b2en proven that the page fetch frequency ratio is not 
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bounded by crc = 2, what bounds, if any, do exist? Hatfield 
iaplicitly raised another question by the statement: 
"as yet we have b22n unable to prove that there is a 
c2placement algorithm using cnly the past history of 
page requests which cannot generate more than twice 
the exceptions with half size pages." 


Tne answers to thes? questions are the subjects of the 


Collowing sections ani chapters. 


3.5 Bounds on the Page Fetch Frequency katio 


It has been shown that the page fetch frejyuency ratio 
can exceed r= 2, put just now bad can it get? Of equal 
Laporctance, what factors influence this bound? fTfhese 


questions will be discussed in this section. 


5.5.1 Cyclic Paye Traces 


Figures 16 and 11 represent page trace simulaticns for 
two sets of demand-fetch LRU-removail two-level storage 
systems with primary store sizes |M!{=2 and jM? j=3, 
caspectively. In both cases, it can be observed that the 
page trace simulated is cyclic with a repeated pattern, Pc. 
In Figure 10, the page trace consists of the repeated 


pattern: 
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As sean by Ss 


) P = a, D, 5, Cy, by a, a, by Cy, 
o {P} = 12 

° Q = { a, b, c } 

* $21 = 3 

e yMty = 2 

e LRU Removal 


As s2en by Ss? 


° P = at, bt, ct, c~, b7, an~, at, 
® }P] = 12 

bs Q = { at, any, bt, D7, ct, c~ } 
» 42] = 6 

e {M24 = & 

e LRU Removal 


Simulation 


transient steady~st 


ate 


j<———- cycle ———-| <> ~scycle ——=/j 


Page fraces. 42° Dec .ch br ata bt ct ce 

Fetch: * * * * * 

Mt contents: a b cc bp a a b cee 
a b b c b b a b ob 

st 

Fetch: x * k * *€ *- &k k & 


M1 Tontents: at bt ct s- bo am at bt ct cm 
at b*t ct cy bum an~ at bt ct 


b- 


a’ 


For the steady-state cycle: 


Results 
F = 6 e F = 2 
F* = 12 ° F' = 6 
crc = 12/fo = 2.9 e® /C/ = 6/2 


Figure 10. 
Cyclic Page Trace with |M*] 


= 2 
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whereas Figure 11 repeats the similar pattern: 


Po = at bt ct dt d- cr b- am 


59.5.2 Steady State Cysiic Paye Traces 


Let us consider Figure 10 first. The page fetch ratio, 
Lt, is 2.9 in this case. AS noted earlier, the page trace can 
b2 suodivided into an initial transient stage, Pt, witha 
high page fetcn frequency followed by a steady-state stage, 
PS, with usually a lower page fetch freguency. In Figure 10, 
the first Pc cycle contains the entire start-up transient 
stage and completely filis all the available space in Mi, 
Thus, the seconi rpc cycle represents the start or the 
steady-state stage. Furthermore, since the content and page 
ocd2ring Of M! is exastly the same at the end of the second 
cycle as they were at the beginning of that cycle for both S 
and st, the page trace cycle, Pc, can be repeated 
continuously with exactly the same results each time for 
page fetch requests ani M! contents. If /ry is defined toa be 
the page fetch £requency ratio for the first steady-state 
p2riod, Pc, of a cyclic paye trace, (Pc)*, /r/ is also the 
page fetch freyuency ratio for the entire steady-state 
portion of the page trace defined by the regular expression: 

P= ptePs = Bte(Pc)* 

AS the length of the page trace, |P|, becomes large in 


SoMmparison with the length of the transient stage, |Ptj, the 
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overall page fetch frequency ratio, cr, asymptotically 
approaches the value of the steady-state cycle page fetch 
frequency ratio, /r/. In Figure 10, sry = 3.0, thus r will 
iacrease from 2.0 towards 3.0 as the page trace is 
lengthened by continually repeating the pattern Pc. linus, 
the page fetch trequency ratio, I, for the page trace 

P = Cae bt ct cm b- am )* 


is bounded by 3.0 when |{M*q{ = 2, 


A similar situation is illustrated in Figure 11. In 
this 2xample, r = 2.23 and /c/f = 4.0. Thus, the paye tetch 
treguency ratio, r, for the page trace 

P= ( at bt ct dt d- ec b- am )* 
15 bounded by 4.0 whan  {M!t{ = 3. By generaliziny these 
2xasples, we arrive at Theorem 3 and Corollary 3a. 
THEOREM 33 

For any two demand-fetch LRU-removal two-level storaye 
systems, S and 5", with page sizes N and N*=N72 and 
primary store sizes |M4j and |M14*=2jM!'], respectively, 
there exists a cyclic paye trace, F = (Pc)*, where j{Pcj 
= 2 (iM*]+1), such that the steady-state page fetch 
frequency ratio, /r/, equals {M4j+1,. 

Proof: 


(See below). 
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(£9) 


As seen by S: 


° P = a,d,c,1,d,c,b,a,a,b,c,4,d,C,b,a 
eo 4P} = 16 

*. Q = { a, b, c, d } 

e 4JQi = 4 

e qm] = j 

°e LRU Removal 


As seen by s! 


° P = ar,b+,ro+,d*+,d~,c~,b-,a~,at+,b*+,ct,d+,d—,c-,b-,a- 
e  |P}] = 16 
° Q = { at, a, b+, b-, ct, cy, d+, d- } 
* 12) = & 
e 1M] = 06 
° LRU xemoval 
Simuiation 


== Se SS eee 


transient steady-state 
|< sSOSycle — | <———- cycle ———"—> 
Page frace: at be c+ dt d- c b- am at bt ct dt d- cm b- amé 
S) 


Patch: «x * * * * * * 
41 Contents: a b c ddcoebaoao»pceodaqeeopia 
aoc c dco pb ba bp c c adc ob 
a b b b ad@¢eecceaibp b b a Cc 

woes. 
Fetch: * * * * * * * * * * * * * * * * 


M4! contents: at* be ct it d- cm bo am at bt ct dt a- cmH E- am 
at pt ct dt d- cm b- am at bt ct dt d- cum b- 


a* b+ ct d+ d- cm bm am at bt ct 


a sane 2) 


Results 
For the steady-state cycles 
° F =7 ° PF = 2 
° F' = 16 * F' = 8 
* c= 6/7 = 2.28 © fxr/ = 8/2 = 4.9 
Figure 11, 


Cyclic Page Yrace with |M!] = 3 
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COROLLARY 3a: 
For any two demand-fetch LRU-removal two-level storage 
systems, S and S*', witn page sizes and N'=N/2 and 
primary store sizes |M!j and |M%]'=2{M4|, respectively, 
there exists a cyciic page trace, F = (Pc)*, where |Pc] 
= 2(/M4]+1), such that the overall page tetch freguency 
catio, CC, asymptotically approaches the bound |M'#{+1 as 


[Pj approaches infinity. 


em ee ee me mem ewe ee wee ee eee ee ae ee we re ee ne eae ee ee we we ee ee ee ee ae ee ee ee ee ee ee ee ee eee ee 


5.5.3 Proof ot Theorea 3 


5.5.3.1 Notation and Properties 
Assum2 a fixed page size N and primary store of size S11, let 
n = tne number of pajyes in Mt (i-@e, Hh = [MP] = StyN). It 
has b2en shown by Mattson et al [€3] that a demand-Lretca 
LRU-removal algorithm has the following properties: 
Pi. If M! is initially empty, it fills with the first 
nh distinct pages referenced by the trace, 
P2. At any tim2 t, Mt contains the n most recently 
referenced distinct payes. 
P3. a) LRU satisfies the inclusion property 
ME(1) C M2(2) C eee C M1 (mM) 
where M!(1) means the contents of M! aif n=t, 


etc, 
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b) At any time t after M! has become filled, there 
is a strict removal ordering referred to as the 
LRU stack 

S = { S(1), S(2), woe, S{n) } 
where 
s(i) = M2 (i) - Mt (i-1) for i = 1, 2, eee, 


and s(n) is the page to be removed next. 


5.5.3.2 Definition 3-a: 

For any integer n, let us consider a page trace, P°, 
consisting of the repeated pattern, Pc®°, of length {Pc%| = 
2(n+1) 

Pp? = pc°[n]* 
where 
Po? nj = { Pco®(1), Po®(2), wae, PoP(2nt+1), Pco®(2n+2) }. 


[The Ps9(i)s are detined as follows; 


Pc? (i) 


fi. for i= 1, anes n+1 
4ne+5-21 for 1 = Nel, evooy 2NtF2 


Ht 
ht 
' 
t 


Thus, for na 


Pc9{ 2) = { 0, 2a 4, De 3, 1 } 


Pet 2] = (9, 2, 4, 5, 3, 1, 0, 2, 4, 5, 3, Ty woe 3} 


Tae cyclic page trace pattern, Pc°{nj], is used to define 
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corcespondiny cyclic paye trace patterns, Pcj nj] and Pc'jn], 
for S and S*, respectively. These are defined as tollcws -- 


For a given value of n and i = 1, 2, oes, 2nt2 


Pc(i) = integer, Pc%(i) /2] 


{integer[ Pc® (1) /2})+ if remj Pc® (i) /2 j=0 
Pc (i) 


(integer[ Pc? (i) /2]))- if rea[ Pc® (i) /2 j=1 
Thus, for n = 2 7-7 
P(2] = { 0, 1, 2, 2, 1, 0, DO, 1, 2, 2 14 De woe } 
PAST = 6-0 Wty Sh, B=. T9004. 1 yp 28 25 1. CF; 
aoe } 
Wea can see that these page traces are identical to the page 
traces of Fiyure 8 with appropriate relabeling (i.e., a=0, 


b=1, c=2) r 


5.5.3.3 Lemma 3-b: 
The page references of the set 
{ Pc(1), eve, Pc(ntl) } 

are distinct. 
Proof: 

Based upon the definitions cf Pc°{n] and Pc[n], we see 
that 

For i = 1, see, ntl 


Pc(l) = integer; Pc® (1) /2]) 


integer[ 2 (i-1)/2] 


iT) 


integer(i-1] 
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= i-1. 
Thus, each value ot Ps(i) for 1 = 1, .»., n+1 is distinct. 


QED. 


52523.4 Lemma 3-c: 
The page references of the set 
{ Po(nt2), «oo, Po({2n+2) } 
are distinct. 
Proof: 
Based upon the definitions cf Pc°%{n] and Pc[n], we see 
that 


Por i = nt+Z, aeoey 2n+2 


Pc(i) integer[ Pc° (i) /2] 

= integer[ (4n+5-21) /2 } 

= integer[ 2n+2+ (1/2) -i]) 

= 2nt2-1i 
Thus, each value of Pc(i) for Ll = OMté, ooo, 2nt2 is 
distinct. 


UeE.D, 


9.5.3.5 Lemma 3-1: 
At the end of each cycle, Pc{n], of the page trace, 
P[nj, “4 contains the pages, in LRU stack order, 


S®9 = { S°(1)4 eee, S9(N) 3 
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where 
$9(j) = j-1 for j = 1, ses, it 
Proof: 

Since each cycla, Pc{n], cf P{[n] is of length e2ntee 
waich is greater that a, the S° LRU stack consists or tae 
last n paye references of Pcjn] in reverse order by property 
P2, P3, and Lemma 3-c. Thus, 

S°({j) = Pe (2n+3-)) 
such that 
$9(1) = Pc(2n+2), s9(2) = Pco(2nt1), woe, S°(n) = Pc(nt3). 
@hen j takes on values { 1, w.+2, n}, 2n+3-j takes on values 
{ 2n+2, «2+, n+3 3}, Thus, for j = 1, «++, nm and pased upon 


Lemma 3-c: 


39 (5) Pc (2n+3-j) 
= 2n+2- (2nt+3-j) 
=. j-l. 


QeE.D. 


5252326 Lemma 3-2; 

Siven a demand-fetch LRU-removal two-level storage 
system, S, with page size N, primary store size 5S! 
Sontaining n=S!/N pages, the paye fetch function, F, 
cesultiny from each steady-state cycle, Pcl[uj, of the 
page trace P uas the value 2 (i.e., Fi Pc{n]j=2 during 


steady state). 
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Proof: 
Let us subdivid2 the Pc{[n] cycle, which is of length 


2ont+2, into four regions as follows: 


Reyion 1: Pcl = { Pc(1), eee, Pc(n) } 
Region 2:3 Pc® = { Pe(n+1) } 
Region 3: Pcs = { Pc(nt2), see, PC(2nt+1) } 


Reyion 4: Pc* = { Pe(2nt2) }. 
and compute the number of page fetches in each region, Fi, 
F2, F3, F*, respectively. Since the page trace regions are 
concatenated, the paje tetches are cumulative, so we know 
that 
P= Fl + Fe + FF + PY, 
R2y1on 13: Pct = { PC(1), «22, Pc{n) } 
From Lemma 3-b, we know that 
Pc({i) = i-1 L = 1, eee, ntl 
and trom Lemma 3-d, w2 know that at the beginning ot each 
cycle 
$9(j) = j-1 J = 1, «oo, Ne 
the page references { PC(1), wee, Pc(n) } are actually the 
s2quence { 9, wee, n-1 } which is identical to the contents 
of M!' at the start of the cycle, S°. Therefore, no paye 
transfers are reguirei although LRU stack reordering may 
occur. (F!=0). 
R2gloa 23 Pc@ = { Pc(nt1) } 
Page reference Po(nt+1) is page n which is net contained 


in S® nor loaded during region 1 (in fact, no pages were 
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fetched during region 1); thus, a page transfer is required 
(F2=1). Using similar techniques as in Lemma 3-d, since each 
reference of Pc! is distinct, the LRU removal stacx at this 
point is 
S = { S(1), eee, S(M) } 

where 

s(j) = Pc(n+1-j) j= 1, eee, De 
Page s(n) is selectei for removal, this is actually page 
Ps (n#1-n)=Pc(1)=0. Thea new LRU stack ordering becomes 

S(j) = Pe(n+2-j) Soa Typ see Ds 
xegion 3: Pc3 = { Pc(nt2), esae,g PC(2nt+1) } 

The page references { Pc(nt2), «wee, Pe(2nt+1) } are 
actually the sequence { n, «s., 1} @S Snownh in the proof of 
La2mma 3-b. The LRU stack ordering immediately prior to 
referance Pc(nt+2) is 

$99 = { S{1), eee, S(n) } 
which is actually 
{ Ny over 13} 
Since it has been shown earlier that at reference Pc(nt2) 
s(j) = Pc(nt+2-3) 4° ee Ay: wep is 

Thus, as in region 1, every page referenced is already 
contained ia SM! and there are no page transfers required 
(2 3=0). 
R2gion 4: Pc* = { Pc(2nt2) } 

Page reference Ps(2n+2) is actually page 9. This paye 


WaS not contained in 59°, thus a page transfer is required 
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[herefore, we can conclude 


Fl; Pct] + F2{Pc2] + F3{Pc3] + P4[Pc4] 


" 


P[ Pc[n]] 


0+ 1+ 9 + 1 


2e 


QED. 


5.5.3.7 Lemma 3-£; 
siven a demand-fetch L&U-removal two-level storage 
system, S*, with page size N*'=N/2, primary store size 
_M4]) containing 2n=[M!]/(N/2) pages, the page fetch 
function, F', resulting from each steady-state cycle, 
Pc'{nj, of the page trace P* has the value 2nt+2 (i.e., 
F'( Pct(najj=2n+2 jiuring steady state). 
Proots: 
The proof follows directly from the definition of P', 
tne LRU properties, and the previous Lemmas. 
e Bach page reference in the cyclic pattern Pce'{nj is 
distinct. (This can b2 easily seen from the definition or 


proven in a Similar manner to Lemmas 3-b and 3-c). 


° Bach cycle is 2nt2 references lcng. 
° At any time t, paye reterence P(t) = P'(t-2n-2). 
° fhe primary store, M!, can hold 2n pages in 5S! siace 


N'=N/2. 
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. Since the cyclic pattern only repeats after 2n+2 steps 
and M! is only 2n pages large, M1! always holds the last 2n 
page references (since they are distinct). 

° Thus, at any time t, page reference P(t) wili aot 
correspond to any paje currently in Mt (i.e., 4! holds 
ceferences { Pf(t-1), «.., P'(t-2n) } and PP! (t)=P' (t-2Zn-2) 
is not in that set). AS a result, a page fetch is reyguirea 
for every paye reference. 

. Since there are 2nt+2 page references per cycle, there 
ace 2n¢2 page fetches required per cycle. Thus, F'=2n+t2,. 


QeE.D. 


5.5.3.8 Theorem 3: 
For any two demani-fetch LRU-removal two-level storaye 
systems, S and S*, with page sizes N and N=N/2 and 
Primacy store sizes |[M!'j'=2jM!j], respectively, there 
exists a cyclic page trace, P=(Pc) *, where 
[Pcf=2(({M4] +1), Such that the steady-state paye L1etch 
frequency ratio, /r/, equals |M4|+1, 
Proofs 
This proof follows trivially from Lemmas 3-e and 3-t. 
W2 koow that for each steady-state cycle ot S, F=2 (Lemma 
3-e). Also, for each steady-state cycle of S', F=2nt2 (Lemma 
3-f). Since the paye fetch trequency ratio, r, is detined as 


f'/£ or (F'/|P|)/(F/1P]) which equals F'/F, we tind that in 
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steady-state 
(Uf = F'/F = (2n42)/2 = nt, 


QeEeD. 


5.5.4 Comments on Theorem 3 


The above results expose another facet of the page size 
anomaly. AS the size of the primary store, M4, iS increased, 
the overall page f2tch frequency ratio as stated in 
Corollary 3a also incr2ases,. This means that tne larger the 
pcoigacy store that you nave, the more "dangerous" the page 
31z2 anomaly becomes. For example, in a two-level paging 
system based on deviztes 2 and 4 frem Table 1, {M*4 = 128 
pages and N = 4096 bytes, if the page size is decreased by 
half to 2048 bytes, it is possible that tne page fetch 
treguancy would increase 1429-told (a 12,800% increase in 
paglay activity!). Of course, one would assume, or at least 
hope, that such pathological page trace patterns would be 
very rare, but we know that they can exist. It is 
Lateresting to note that th2 pathclogical pattern shown 
above (@.J., a* b* ct c- b- a) corresponds to the expected 
ceferances of nested subroutine calls (i.e., subroutine a 
calls subroutine b which calls subroutine c, etc., and each 
subroatine, of course, returns te its caller). This is also 
.tcue of other stack-like program ccnstructs, Such highly 


wodular program design is quite typical and, furthermore, is 
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often explicitly encouraged. In view of Hatfield's finding 
where the overall c exceeded 2.0 in many programs, it 1s 
reasonable to assume that there were probably regions in 
which rwas quite small, possibly below 1.0, which were 
countarbalanced by regions with very high values of cx. At 
preseat we do not have this particular intormation 
available, but if it were true, performance could be greatly 
iaproved by eliminatiny the high cr value rcegions. This 


problem will be discussed in the next section. 


5.5.5 Bounds for FIFO Removal Algorithm 


Theorem 3 applies to LRU removal algorithms and many 
Other removal algorithms, although these other cases will 
not b2 explicitly proven in this thesis. It is interestiny 
to conSider whether the result of Theorem 3 applies to the 
FIFO removal algorithm, Unfortunately, due to the 
peculiarities of FIFO, a simple generalizable cyciic paye 
trace pattern has not been found. But, isolated examples 
have been found, as illustrated in Figure 12, that show that 
it is possible for rc to exceed [|M!{¢1. This result is stated 
in Theorem 4, Based upon other examples, it is conjectured 


that the cr, when FIFO removal is used, may be as high as 
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(£19) 
parameters 
As seen by 8S: 
e P =a,c,a,b,b,c,C,a,a,b,b,Cc,c,a,a,b,b,c,c 
e |Pi =19 
° Q ={ a, b, c } 
°® {Q) =3 
e jMiy=2 
® FIFO Removal 
As seen by S*3 
*® Pp =at,c-,a-,b+,b-,ct,c-,at,a-,bt,b-,ct,c-,at,a—, b+ ,b- ,c+,c~ 
e yPy =19 
7 Q ={ at, a~, bt, b-, ctr, cmH } 
© }Q1 = 
® {M2]=4 
e FIFO Removal 
Simulation 
steady-state 
|e _ transient —-—~ |< _ cycle ————————————————— 
Trace: at c~ am bt b- ct cm at am bt bm ct cm at am~ bt Dm ctr cm 
iS) 
Patch: * ¥* * 53 = : 
Mis a c ¢c b b b ba aaa'icqccoeeob ib b&b b 
a acccocop b b b a a aA aC EF CCS 
Sf 
Fatch: * * * #*€ © * * * & # * ke & He ek KF ex 
Mi: at cm am bt br ct o- at a- bt bom ct cum at anm~ bt bm ct cm 
at cm am bt b- ct cm at am bt b- ct cm at arm bt bm ct 
at cm a~ bt bo ct cm at arm~ bt b- ct cm at am bt D- 
a* cy- an~ bt b- ct cy at ar-~ bt bm ct cm at arm bt 
_—e same sb cette 
K2sults 
For the steady-state cycle: 
° F = 6 . F = 3 
e Ft = 19 e Ft = 12 
Ld Cr = 19/6 = 3.10 a SU/ = 12/3 = 4.0 
Figure 12. 


Cyclic Paya Trace with FIFO Removal 
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THEOREM 4: 

For any two demand-fetch FIFO-removal two-level storage 
systems, S and S*, witn page sizes N and N*=N/2 and 
sertain primary store sizes ,{Mt{ and jMtj'=2jM'], 
cespectively, there exists a cyclic page trace, P = 
Pte(Pc)* where  jPcj] = 2({M24+1) (¥M24), such that the 
page fetch frequancy ratio, r, exceeds |Mtj+1. 

Proof: 


By example (Figure 12). 


22 we ee ww we Ow wwe @ wm we wee ee ee ee ee ew ew we wee we wwe ww ee ee oe ee wwe ee we wee 
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CHAPTER 6. 


SPATIAL VS. TEMPORAL LOCALITY MODEL OF PROGRAM BEHAVIOR 
6.0 Introduction 


Early in this thesis it was explained that a major 
rationale for multilevel storage systems is based upon the 
Principle of Locality. Unfortunately, locality is stilia 
poorly understood, or at least controversial, phenomenog,. In 
this chapter some novel viewpoints and insiyhts wili he 


presented, 
6.1 Types of Program Reference Locality 


Let us consider two extreme torms of program reference 


locality which will be called temporal locality and spatial 


6.1.1 Temporal Locality 
If the logical addresses { a4, a?, ... } are reterenced 
during the time interval t-T to t, there is a high 
probability that these same logical addresses will be 
referenced during the time interval t to t#T. 


[This behavior can be rationalized by program constructs 
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such as: loops, trequently used variables, and 


frequently used subroutines. 


6.1.2 Spatial Locality 
If the logical address a is referenced at time t, there 
is a high probability that a logical address in the 
range a-A to atA will oa referenced at time ttl. 
This behavior can be rationalized by program constructs 
such as: sequential instruction sequencing, and linear 


data structures (e.g., arrays). 


6.1.3 General Locality 


[The definitions of temporal and spatial locality above 
are guite extreme. Usually we consider only the general 
Spatiotemporal properties and define locality as: 

Locality 
If the logical addresses { at, a?, se. } are reterenced 
during the time interval t-T to t, there is a higa 
probability that the loyical addresses in the ranges 
al-A to atltA, a2-A to a@tA, .22, Will be referenced 
luring the time interval t to ttT. 
it is important to recognize that temporal locality and 
spatial locality are indeed the underlying phenomenon and 
that the "general locality" is merely a simpliftyiny merging 


and blurring of these basic concepts. 
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#e cau begin to understand the factors causing the page 
siz2 anomaly by stadying now the various conventional 
removal algorithms handle temporal and spatial locality. In 
particular, we see, that whereas temporal locality policies 
ace yglven 2xplicit attantion, spatial locality policies are 
usually handled implicitly and subtlely. The "least recently 
usei", LRU, cemoval algorithm, for example, is very much 
concerned apauk: the temporal aspects of the pregram's 
reference pattern, The Spatial aspects are handled as a 
by-product of the rcact that the demand fetch algorithm must 
load an entire page (i.@., a spatial region) at a time and 
LRU c2moval decisions are based upon these pages. With these 
thoughts in mind, w2 can see that decreasiny page size 
causes the conventional storage management algorithms to 
increase their sensitivity to temporal locality and decrease 
their sensitivity to spatial locality. Increasing page size, 


of course, results in the reverse effect, 
6.3 Locality in Actual Programs 


Many of the t2chnigues for improving the locality 


behavior of programs, suca aS the method or automatic 
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peogram restructuring by sectcr (subroutine) reordering 
dascribed by Hatfield and Gerald [47], result in both 
Lacreased temporal and spatial lccality. But, it seems that 
the reordering technique does, in fact, significantly tavor 
spatial locality since it was noted [47] that: 
"the better ocjerings not only concentrate 
appropriate sectors into pages, but these pages also 
naturally cluster into larger units that satisty 
Nearness requirements on the page level - and 
cluster better than dothe pages ot the other 


orderings ..e. clustering sectors into pages also 
clusters payes into larger units." 


An effective multilevel storage management system must 
tak2 poth temporal and spatial locality into consideration. 
AS we have seen from both Hatrield's and Seligman's results, 
neglecting spatial locality can have disasterous results. 
Any given program, or portion of a program's operation, can 
uave its reference lotality characterized by the two-by-two 


matcix: 


TEMPORAL 


Rrra Oo wW 


wuadrant 1, \low-temporal and low-spatial locality, is 
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dafinitely undesirable for operation in a multilevel storage 
system, There have b2en numerous algorithms and programmer 
training techniques developed, as mentioned above, to 
minimize the number ot programs with these poor locality 
Characteristics. Quadrant 4, high-temporal and high-spatial 
locality, has traditionaly been the regicn of pest 
performance and is usually the objective of good program 
d2siga. Unfortunately, it is not always possikie or 
convenient to design programs Which attain both high 
temporal and high spatial locality; thus, we find many 


poojcams operating in guadrants 2 or 3. 
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Storay?@ management techniques are needed which precvide 
far more flexibility and robustness for balancing the 
systea's sensitivity to temporal and spatial locality. These 
algorithms must explicitly consider the spatial locaiity of 
a program. The tuple-coupling apprcach, described in the 
n2xt chapter, is one such technique. It takes advantaye ot 
the tamporal locality and compactness possible with small 
pages characterized by quadrant 2 behavior, yet it adjusts 
t> tne spatial locality and clustering characterized by 
quadcant 3 behavior by simulating the removal policies 


associated with large pages, 
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6.6 Comment on the Paye Size Anomaly 


With this insight, we can now see that the page size 
anomaly is not really 2ven a function strictly of paye size! 
iastead, it is an issue of locality, temporal versus 


Spatial. 
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CHAPTER 7. 


SPATIAL REMOVAL STORAGE MANAGEMENT ALGORITHMS 


7,0 Introduction 


As stated earlier in this thesis and noted by Hatfield, 
a cenoval algorithm that would limit the page fetch 
frequency ratio, cr, to 2 would be very desirable. In this 
section a technigue, called the “tuple-coupling approach", 
is described which, when used in conjunction with 
conventional removal algorithms, such as LRU or “FIFO, 


juarantees that r will not exceed 2, 


The basic concept behind the tuple-coupling approach is 
extremely simple. First, the two pertions, pt and pr-, ot 
aach original larger page, p, must be identifiable (i.e., 
the set of pages of S' are viewed as a collection of 
2-tuples). Second, the removal ordering policies aust be 
applied to both elements of a tuple (i.e., the tuples are 
coupi2d in reyard to ordering decisions) such that a page p+ 
or p- of 3* is never ramoved unless the corresponding page p 


of S would also have been removed from M', The particular 
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implementation of this approach may vary slightly depending 
upon the cemoval alyorithm, @.g., LRU, FIFO, etc., that is 
to be used. Any removal algcritam ta which the 
tuple-coupling approach can be incorporated is said to be 


"tuple-couple-able", 
7.1.1 An Example of LaU Tupie-Coupling 


Figure 13 illustrates the application ot the 
tuple-coupling approach to the LRU removal example 
previously shown in Figure 9. It shculd be noted that, in 
this case, C has indeed been limited to 2 although it had a 
value of 2.2 when normal LRU removal was used. The reader 
Should carefully compare Figures 7 and 11 to understand how 
the tuple-coupling approach affects the removal alyoritha. 
The M! contents are identical, of course, for S in both 
examples, but there are subtle differences in M! contents 
for S*, Each state of M! contents is marked, 1 to i1, in 
Figure 13 for reference purposes, Notice that in this 
implementation of tuple-coupling whenever both halves of a 
paye, pt and p-, are in M!, they are always adjacent in the 


M! ordering; compare this with Figure 9. 


At page trace step 3 we can see the first difference 
batwean Figures 7 and 11. Page a is reterenced and must be 


teatched in Figure 9, it is then placed at the top of the M! 
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Parameters 


As sean by S: 


° P =a, b 

@ {Py = 11 

° Q = { a, 

. 1QO1 = 3 

e {M44 = 2 

® LRU Removal 
AS se2n by S*: 

e P = at, 

°e «Py = 11 

° 6) = { at, 

ed 121 = 6 

° {M2} = 4 
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LRU Removal with Tuple-Ccupling 


Simulation 
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Page Trace: at bt 
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Patch: oe 
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Example of LRU Removal with Tuple~Coupling 
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ordering waich becom2s a-,bt,at. On the other hand, in 
Pigure 13 at step 3, it is noticed that at was already in 
Mi. Thus, when a~ is placed at the top of the M! ordering, 
at is coupled to it resuitiny in the ordering a-~,at,bt,. At 
page trace step 7 of Figure 13 we see another interesting 
example of tae tuple-soupling approach. At the previous step 


the ordering was 


when the reference to bt is made, there is nc need to 
initiate a fetch since bt is already in M!. The M*# ordering 


then becomes 


Since LRU requires that the most recent rererence move to 
tae top. Under this tuple-coupling scheme, b- is also moved 
toward the top of tne ordering te continue to be adjacent to 


bt, 


7.1.2 Implementation of the Tuple-Coupling Approach 


It is important to note that there are often various 
ways to implement tuple-coupling. In particular, in the LRU 
tuple-coupliny alyorithm described above, the 2-tuples, 
wnenever both portions were in M4, were arranged to be 
ajijacent in the M! rcenoval ordering. The reyguirement that 


u2itharc portion, p* or p-, of a tuple in S' be removed 
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unless the corresponding page of S would have been removed 
can be accomplished in other ways. For example, the LRU 
removal stack can be left in its normal ordering, as in 
Figur2 9. In this case, when it 1S necessary to remove a 
page Erom S* the bottom page is not necessarily the correct 
choice to satisfy tuple-coupling. There is an algorithr 
which can scan the LRU stack and select the correct paye for 
ramoval (in fact, it will select, of ccurse, the same page 


salected by the algorithm illustrated in Figure 13). 


7.1.3 An Example of FIFO Tuple-Coupling 


It is interesting to consider the effect of 
tuple-coupling upon FIFO removal. Figure 14 illustrates the 
application of the tuple-coupling approach to the FIFO 
c2moval example previously shown in Figure 6. Once again, 
the page fetch freyuency ratio, rc, which originally was 2.75 
has indeed been limited to 2. The example of Figure 14 does 
not fully illustrat2 ali the interesting aspects of 
tuple-coupling upon FIFO removal. In particular, if page pt, 
foc example, is referanced in a page trace and it was not 
already in Mt, it must be fetched. The M! contents are 
ceordared as follows: 

1, If p- is not currently Mt, ¢* is placed at the top 

of the FIFO ordering, 


2. If p- is currently in M!, ft is placed immediately 


As seen by S: 
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before p- in the logical FIFO crdering 
p-'s relative ordering remains unchanged. 
The ceason for the second part of this rule can be seen from 
the normal FIFO ordering rule which places a page p at the 
top only if it were not already in MM, If it were in Hi, it 
remains at its previous ordering position. Under 
tuple-coupling, this rule applies jointly to the (p*t,p~) 
taple as stated above. The reader is encouraged to work 
tarough the example of Figure 10 using the tuple-ccupling 
approach to illustrate this FIFO ordering phenomenom, The 
effect of the tupla-coupling approach is summarized in 


Theorem 5, 


(th5) 
THEOREM 5: 


For any two demand-fetch two-level storage systems, 5 
and S*, with pag2 sizes N and N'=N/2, respectively, the 
use of the "tuple-coupling" approach for S* in 
conjunction with a remcval algorithm that is 
"tuple-couple-able" is sufficient to guarantee that the 
page fetch frequency ratio, r, cannot exceed the value 
2 for all possible page traces, P, 

Proof: 

(See below), 


eee ee ee eee ee ee ee ee ee em ee ee rr ee ee ee oe ee ee ee ee ee ae ae eee 


Storaje dierarchy Systems 133 


7.12% Proof ot Theorem 5 


As described earlier, when an adress trace, A, is 
appliad to storaye systems S (with page size N) and S* (with 
page size N'=N/2), it can be represented as page traces P 
and P*, respectively. At time t!, let us consider a specitic 
aidress reference, a, whose corresponding page references 
are p (in S) and pt (in S'). In processing this reference 
there are four possibl2 fetch actions in S and S* depending 
upon the current content state of primary store, M1; 
erceCt 


Seeks. ee 


‘pad ee ie (") 


Page PAS) 


fimeee ee ie 
ee ee eee | ee 


kecall that the paye fetch frequency ratio, c, eguals 


F'/F. In states 1 and 4 the same action (i.€., no pfage fetch 
ia 1 and a page fetch in 4) occurs in both S and S'*, the 
occurrence of these states cause r to tend towards 1. in 
State 3, a page fetch 1s required in S but not in S', this 
Situation, if frequent, wili cause r to decrease toward 
zero. This is usually the intended result of reducing page 
size. Only state 2, in which S* alone requires a page retch, 
contributes to an increase in rc. Thus, we will concentrate 


Qur analysis on this particular situation. 


Storage Hierarchy Systens 134 


Since state 2 reyuires that page p be in M! at time ti, 
if we scan the address trace backwards, there must be sore 
previous reference time t? that caused page p (in S) to be 
tetched into M! (this may have been the only previous 
reference to p or the page p may have been fetched and 
removed many times). At time t2, there must also be a 
correspondiny referens2 to either p- and pt of S', These two 


cases will be considered separately: 


Pp! = eve PDP” eve pt 


tt 
tt 


eee t2 we. t! 

This case merely illustrates the fact that it can 
reyguire two page fetches (for pt and p-) in S* to fetch the 
Same amount of storage as page p in S. If this were the only 
case tor state 2, cr would never exceed 2. 


Case 2:3 P = eee P oes P 


las) 
u 


oe Pt eee pt 


tt 
ul 


eos t2 woe tit 

In this case we see that subsequent to reference t? 
page p of S and page p+ on S* must be in M!, Yet at time t} 
page p of S is still in Mt but page pt of S* is not, Under 
these circumstances r can certainly exceed 2, merely makiny 
p> the next referents will account for 3 fetches in 5S! 
compared to 1 fetch in S. Furthermore, it is possible that 


the ceterences between t® and tt could be repeated to 
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continually cause fetches for pt in s! without any 
corcespondiny fetches required in S. Thus, we see that this 


is precisely the situation that ailows r to exceed 2, 


Under closer analysis, we see that this situation 
reguires that in S' p+ be removed frem M' between t2 and t} 
whereas in S p remains in M!. In other words, this general 
Situation can only occur if at some time t, p* or p- of S! 
is selected for removal from M! and the corresponding paye p 
of S is not also removed from M1, But, the tuple-coupling 
algorithm (see page 125) is “such that a page pt or g¢~ ot S! 
is never removed unless the correspondiny page p of S would 
also have been removed trom M!", Thus, the tuple-coupling 
2liginates the possibility of case 2 and therefore 
guarantees that r cannot exceed 2. 


QeE.D. 


Clearly, the tuple-coupling approach has an influence 
upon tne overall etfectiveness of the basic rewoval 
algorithm being used and the benefits of the smaller page 
siz2. It is obvious that there are certain reference 
patterns (with c iless than 2) for which tuple-coupling 
laccreases the value of r. On the other hand, it can be 


Shown, as a Simple exercise for the reader, that the example 
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of Figure o retains its low page fetch frequency ratio of 
3.5 even when tuple-coupling is used. In tact, 
tuple-coupling may oftan result in the "best of both worlds" 
by placing a bound on the page fetch frequency ratio, r, for 
high c regions without interfering with the performance of 


Originally low r regions. 


4 program's reference behavior in S‘', during a short 
interval of its operation, may be characterized by three 
creyioas based upon the value of the page fetch freyguency 
catio, cr, when tuple-coupling is not used: 

1. Sparse reference - small rc (e.g., less than 1). 

2. Moderate refarence ~ moderate rc (@€.g., between 1 
and 2). 

3. Danse reference - high r @2.g., greater than 2). 

In th2 sparse reference region, it is unlikely that both 
pottions, pt and p-, of a page, p, will be in “4M 
simultaneously; thus, the tuple-coupling will have ainimal 
affect upon performance. In the dense reference region, we 
havea already sean that tuple-coupling prevents extreme 
values of c. Based upon some recent, though limited, 
ugasucements, it appaars that in the moderate reference 
rejion tuple-coupling pertorms about as well as the 


non-tuple-coupled algorithms. 
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CHAPTER 8. 


DISCUSSION AND CONCLUSIONS 


8.0 Introduction 


Efficient and effective storage management is important 
to the development of future computer systems. It has been 
astimated that the storage subsystems account for over 70% 
of the cost of most contemporary installations and, based 
upon present trends, this percentage is expected to 


increase, 


Much more research will te needed before all the 
problems of automatic storage management are understood and 
the obstacles to effective operation eliminated, This 
thesis has solved several open problems and has’ provided 
insight that should lead to the solution of many sore 


problems. 


A detailed discussion of the many ftacets ot stcraye 
management is present2d in Chapter 2. It also contains a 


general discussion of the reguirements which a system must 
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satisty to be etfective for the user. 


In Chapters 3 and 4 a model for storaye hierarchy 
systeus is formalized and an implementation is proposea. The 
systeats design is based upon an crderly and unitors 
treatment of the storage levels. Specific techniques to 
laprove performance, such aS continuous hierarchy, shadow 
storage, direct trdnsfer, read through, store behind, and 
automatic management, are explained. 

In Chapter 5 the "page size ancmaly" is presented (see 
also Hatfield [48]}): 

"The assumption about virtual memory systems that as 
Overhead (time for access and software page 
management) decreases page size should be reduced is 
not always a yood one. Recent experiments indicate 
that larger sizes can provide better performance for 
programs that mak2 highly localized use of memory 
space," 
Tnis ophenomenom is formalized and a bound on the perfcrmance 


is proven, 


In Chapters 6 ani 7 the concept of spatial locality is 
intcoduced and serves as the basis for a new storage removal 
algocithm called “tuple-coupling"™. These concepts are used 
to explain the occurrence ot the “page size anomaly" in 
actual systems, It iS proven that the tuple-coupling 


approach is a sufficient strategy to avoid the occurrence of 
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the “page size anomaly" and it ctifers potential performance 


improvements for the storage Alerarchy systeu. 


The techniques and theorems presented in this thesis 
provile a much more scientifically sound basis for examining 
and designing storage hierarchy systems than wost current ad 
hoc approaches. Although there is still a long way to go, 
development of these formalisms is essential to the 


aivancing of the "science" in Computer Science. 


3.2 Further work 

There are many areas touched on by this work in which 
questions remain, One of the most siyniticant 15 in the 
development and study of other possible "spatial locality" 
cemoval alyorithms in additicn tec the tuple-coupling 
approach studied in this thesis, This iS an entirely wide 


open area. 


Although tuple-coupling is studied extensively in this 
thesis, there are still many unanswered gueStions. dHcw does 
tuple-coupling compare with the class of "stack" algorithms 
studied by Mattson 2t al [603], in particular under what 
circumstances, if any, iS tuple-coupling a stack algorithna? 
Likewise, how does tuple-coupling compare with the 


theoretically optimal replacement algorithm, called OPT [63] 
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ac MIN [12]? On a mor? practical side, how etticiently can a 
tuple-coupling algorithn, or other Spatial removal 


alyorithms, be implemented? 


In ord®r to ascertain specific procf of the utility and 
afficiency of general storage hierarchies, it will be 
necessary to actually construct and measure the performance 
of such aosystem or, at least, perform more extensive 
Simulation analysis. Furthermore, we must develop overall 
projramming techniygues and execution environments that are 
aven more amenable to efficient operation in a storage 


hierarchy systen, 


Many of these questions are curreatly under 
lavestigation, the results will be published later in a AIT 


Project MAC Technical Report. 
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